2024-03-28T15:54:18.191 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-28T15:54:18.191 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-03-28_15:43:50-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7628301 --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} --verbose --name yuriw-2024-03-28_15:43:50-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-03-28_15:43:50-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7628301/orig.config.yaml 2024-03-28T15:54:18.231 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T15:54:18.318 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-28_15:43:50-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7628301 branch: wip-yuri4-testing-2024-03-26-1132-reef description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7628301' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-03-28_15:43:50-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri4-testing-2024-03-26-1132-reef ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: e58b9979847f029f7ab1514b5f5f99b9f03d97f6 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: e58b9979847f029f7ab1514b5f5f99b9f03d97f6 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2024-03-26-1132-reef sha1: e58b9979847f029f7ab1514b5f5f99b9f03d97f6 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6904 sha1: e58b9979847f029f7ab1514b5f5f99b9f03d97f6 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2024-03-26-1132-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: e58b9979847f029f7ab1514b5f5f99b9f03d97f6 targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPSlIhOqcL4HPPLuorlvP91WM/eGDMhSxwDVQxUeX+rJZctwWzPZuqCmEEZMlGLS1+H4aGa0KHzKTPaqWlGLt5A= smithi083.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPEEzb0wEA3mhYdRzG0yLCtBdbNckMpzYgietdNUhBv07DwgXUIwEb937mdiXay9pYyp/gyhuKgZaVKV1FcQxNA= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-28_15:43:50 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-28T15:54:18.318 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa; will attempt to use it 2024-03-28T15:54:18.319 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks 2024-03-28T15:54:18.319 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-28T15:54:18.320 INFO:teuthology.task.internal:Checking packages... 2024-03-28T15:54:18.346 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e58b9979847f029f7ab1514b5f5f99b9f03d97f6' 2024-03-28T15:54:18.347 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-28T15:54:18.347 INFO:teuthology.packaging:ref: None 2024-03-28T15:54:18.347 INFO:teuthology.packaging:tag: None 2024-03-28T15:54:18.347 INFO:teuthology.packaging:branch: wip-yuri4-testing-2024-03-26-1132-reef 2024-03-28T15:54:18.347 INFO:teuthology.packaging:sha1: e58b9979847f029f7ab1514b5f5f99b9f03d97f6 2024-03-28T15:54:18.347 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri4-testing-2024-03-26-1132-reef 2024-03-28T15:54:18.493 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1053.ge58b9979 2024-03-28T15:54:18.494 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-28T15:54:18.500 INFO:teuthology.task.internal:no buildpackages task found 2024-03-28T15:54:18.500 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-28T15:54:18.536 INFO:teuthology.task.internal:Saving configuration 2024-03-28T15:54:18.551 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-28T15:54:18.569 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T15:54:18.594 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-28_15:43:50-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7628301', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 15:49:09.915584', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPSlIhOqcL4HPPLuorlvP91WM/eGDMhSxwDVQxUeX+rJZctwWzPZuqCmEEZMlGLS1+H4aGa0KHzKTPaqWlGLt5A='} 2024-03-28T15:54:18.618 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-28_15:43:50-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7628301', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 15:49:09.914228', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPEEzb0wEA3mhYdRzG0yLCtBdbNckMpzYgietdNUhBv07DwgXUIwEb937mdiXay9pYyp/gyhuKgZaVKV1FcQxNA='} 2024-03-28T15:54:18.618 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-28T15:54:18.625 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-28T15:54:18.625 INFO:teuthology.task.internal:roles: ubuntu@smithi083.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-28T15:54:18.625 INFO:teuthology.run_tasks:Running task console_log... 2024-03-28T15:54:18.719 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3f44603f70>, signals=[15]) 2024-03-28T15:54:18.720 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-28T15:54:18.728 INFO:teuthology.task.internal:Opening connections... 2024-03-28T15:54:18.728 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2024-03-28T15:54:18.730 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:54:18.837 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi083.front.sepia.ceph.com 2024-03-28T15:54:18.838 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:54:18.935 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-28T15:54:18.943 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2024-03-28T15:54:18.960 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2024-03-28T15:54:18.960 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2024-03-28T15:54:19.016 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="CentOS Stream" 2024-03-28T15:54:19.017 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="9" 2024-03-28T15:54:19.017 INFO:teuthology.orchestra.run.smithi007.stdout:ID="centos" 2024-03-28T15:54:19.017 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="rhel fedora" 2024-03-28T15:54:19.017 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="9" 2024-03-28T15:54:19.018 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el9" 2024-03-28T15:54:19.018 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T15:54:19.018 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31" 2024-03-28T15:54:19.018 INFO:teuthology.orchestra.run.smithi007.stdout:LOGO="fedora-logo-icon" 2024-03-28T15:54:19.019 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T15:54:19.019 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://centos.org/" 2024-03-28T15:54:19.019 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T15:54:19.019 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T15:54:19.019 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T15:54:19.020 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2024-03-28T15:54:19.048 DEBUG:teuthology.orchestra.run.smithi083:> uname -m 2024-03-28T15:54:19.066 INFO:teuthology.orchestra.run.smithi083.stdout:x86_64 2024-03-28T15:54:19.066 DEBUG:teuthology.orchestra.run.smithi083:> cat /etc/os-release 2024-03-28T15:54:19.121 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="CentOS Stream" 2024-03-28T15:54:19.121 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="9" 2024-03-28T15:54:19.121 INFO:teuthology.orchestra.run.smithi083.stdout:ID="centos" 2024-03-28T15:54:19.121 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE="rhel fedora" 2024-03-28T15:54:19.121 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="9" 2024-03-28T15:54:19.121 INFO:teuthology.orchestra.run.smithi083.stdout:PLATFORM_ID="platform:el9" 2024-03-28T15:54:19.121 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T15:54:19.121 INFO:teuthology.orchestra.run.smithi083.stdout:ANSI_COLOR="0;31" 2024-03-28T15:54:19.121 INFO:teuthology.orchestra.run.smithi083.stdout:LOGO="fedora-logo-icon" 2024-03-28T15:54:19.121 INFO:teuthology.orchestra.run.smithi083.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T15:54:19.121 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://centos.org/" 2024-03-28T15:54:19.122 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T15:54:19.122 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T15:54:19.122 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T15:54:19.122 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server 2024-03-28T15:54:19.147 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-28T15:54:19.165 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-28T15:54:19.196 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-28T15:54:19.197 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2024-03-28T15:54:19.199 DEBUG:teuthology.orchestra.run.smithi083:> test '!' -e /home/ubuntu/cephtest 2024-03-28T15:54:19.216 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-28T15:54:19.222 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-28T15:54:19.223 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2024-03-28T15:54:19.257 DEBUG:teuthology.orchestra.run.smithi083:> test -z $(ls -A /var/lib/ceph) 2024-03-28T15:54:19.277 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-28T15:54:19.332 INFO:teuthology.run_tasks:Running task kernel... 2024-03-28T15:54:19.353 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-28T15:54:19.354 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-28T15:54:19.354 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T15:54:19.355 DEBUG:teuthology.orchestra.run.smithi083:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T15:54:19.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T15:54:19.372 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-03-28T15:54:19.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T15:54:19.373 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-03-28T15:54:19.430 INFO:teuthology.orchestra.run.smithi083.stdout:5.14.0-361.el9.x86_64 2024-03-28T15:54:19.430 INFO:teuthology.task.kernel:Running kernel on smithi083: 5.14.0-361.el9.x86_64 2024-03-28T15:54:19.430 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum install -y kernel 2024-03-28T15:54:19.431 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-361.el9.x86_64 2024-03-28T15:54:19.431 INFO:teuthology.task.kernel:Running kernel on smithi007: 5.14.0-361.el9.x86_64 2024-03-28T15:54:19.432 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum install -y kernel 2024-03-28T15:54:21.833 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - AppStream 9.0 MB/s | 6.7 MB 00:00 2024-03-28T15:54:21.840 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - AppStream 8.2 MB/s | 6.7 MB 00:00 2024-03-28T15:54:23.300 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-28T15:54:23.591 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - BaseOS 3.1 MB/s | 2.0 MB 00:00 2024-03-28T15:54:24.383 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-28T15:54:26.097 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - BaseOS 3.7 MB/s | 8.0 MB 00:02 2024-03-28T15:54:30.615 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - AppStream 5.4 MB/s | 19 MB 00:03 2024-03-28T15:54:36.632 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.1 MB 00:02 2024-03-28T15:54:37.991 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - Extras packages 79 kB/s | 16 kB 00:00 2024-03-28T15:54:41.423 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-28T15:54:45.350 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 2.9 MB/s | 21 MB 00:07 2024-03-28T15:54:46.474 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 6.1 MB 00:01 2024-03-28T15:54:48.266 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 16 kB 00:00 2024-03-28T15:54:50.200 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-28T15:54:51.129 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 7.5 MB/s | 21 MB 00:02 2024-03-28T15:54:52.323 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-28T15:54:52.324 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-28T15:54:52.406 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T15:54:52.416 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T15:54:52.416 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-03-28T15:54:52.416 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T15:54:52.417 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T15:54:52.417 INFO:teuthology.orchestra.run.smithi083.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-28T15:54:52.417 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-28T15:54:52.417 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-28T15:54:52.417 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-28T15:54:52.417 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-28T15:54:52.417 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T15:54:52.417 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T15:54:52.418 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T15:54:52.418 INFO:teuthology.orchestra.run.smithi083.stdout:Install 4 Packages 2024-03-28T15:54:52.418 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T15:54:52.419 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 76 M 2024-03-28T15:54:52.419 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 123 M 2024-03-28T15:54:52.419 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T15:54:53.126 INFO:teuthology.orchestra.run.smithi083.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 152 kB/s | 66 kB 00:00 2024-03-28T15:54:56.193 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-28T15:54:58.441 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-28T15:54:58.442 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-28T15:54:58.537 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T15:54:58.562 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T15:54:58.562 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-03-28T15:54:58.562 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T15:54:58.562 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T15:54:58.562 INFO:teuthology.orchestra.run.smithi007.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-28T15:54:58.562 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-28T15:54:58.562 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-28T15:54:58.562 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-28T15:54:58.562 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-28T15:54:58.562 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T15:54:58.562 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T15:54:58.562 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T15:54:58.563 INFO:teuthology.orchestra.run.smithi007.stdout:Install 4 Packages 2024-03-28T15:54:58.563 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T15:54:58.563 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 76 M 2024-03-28T15:54:58.563 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 123 M 2024-03-28T15:54:58.563 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T15:54:58.693 INFO:teuthology.orchestra.run.smithi083.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 2.5 MB/s | 15 MB 00:06 2024-03-28T15:54:58.919 INFO:teuthology.orchestra.run.smithi007.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 243 kB/s | 66 kB 00:00 2024-03-28T15:55:00.203 INFO:teuthology.orchestra.run.smithi007.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 9.6 MB/s | 15 MB 00:01 2024-03-28T15:55:00.338 INFO:teuthology.orchestra.run.smithi083.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 4.4 MB/s | 33 MB 00:07 2024-03-28T15:55:00.602 INFO:teuthology.orchestra.run.smithi083.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 3.7 MB/s | 27 MB 00:07 2024-03-28T15:55:00.602 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T15:55:00.602 INFO:teuthology.orchestra.run.smithi083.stdout:Total 9.3 MB/s | 76 MB 00:08 2024-03-28T15:55:01.267 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T15:55:01.624 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T15:55:01.625 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T15:55:01.711 INFO:teuthology.orchestra.run.smithi007.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 11 MB/s | 33 MB 00:03 2024-03-28T15:55:02.303 INFO:teuthology.orchestra.run.smithi007.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 8.1 MB/s | 27 MB 00:03 2024-03-28T15:55:02.306 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T15:55:02.306 INFO:teuthology.orchestra.run.smithi007.stdout:Total 20 MB/s | 76 MB 00:03 2024-03-28T15:55:02.648 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T15:55:02.648 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T15:55:03.000 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T15:55:03.476 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T15:55:03.476 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T15:55:04.541 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T15:55:04.541 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T15:55:05.127 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T15:55:05.746 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-28T15:55:06.045 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T15:55:07.141 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T15:55:07.772 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-28T15:55:07.873 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T15:55:08.787 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T15:55:08.925 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T15:55:10.742 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T15:55:10.876 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T15:55:13.617 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T15:55:13.726 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-28T15:55:15.956 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T15:55:16.002 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-28T15:55:18.470 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T15:55:21.101 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T15:55:47.979 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T15:55:47.979 INFO:teuthology.orchestra.run.smithi083.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T15:55:47.979 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T15:55:48.023 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-28T15:55:48.384 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-28T15:55:48.384 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-28T15:55:48.384 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T15:55:48.384 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T15:55:48.769 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T15:55:48.770 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T15:55:48.770 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T15:55:48.770 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-28T15:55:48.770 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-28T15:55:48.770 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-28T15:55:48.770 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-28T15:55:48.770 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T15:55:48.770 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T15:55:49.104 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T15:55:49.208 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-28T15:55:49.209 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-28T15:55:49.209 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-28T15:55:49.209 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-28T15:55:49.209 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-28T15:55:49.209 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-28T15:55:49.209 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi083.front.sepia.ceph.com, path=None, version=distro) 2024-03-28T15:55:49.210 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T15:55:49.290 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-28T15:55:49.290 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-28T15:55:49.290 DEBUG:teuthology.orchestra.run.smithi083:> sudo rpm -qi grub2-tools 2024-03-28T15:55:49.324 INFO:teuthology.orchestra.run.smithi083.stdout:Name : grub2-tools 2024-03-28T15:55:49.325 INFO:teuthology.orchestra.run.smithi083.stdout:Epoch : 1 2024-03-28T15:55:49.325 INFO:teuthology.orchestra.run.smithi083.stdout:Version : 2.06 2024-03-28T15:55:49.325 INFO:teuthology.orchestra.run.smithi083.stdout:Release : 68.el9 2024-03-28T15:55:49.325 INFO:teuthology.orchestra.run.smithi083.stdout:Architecture: x86_64 2024-03-28T15:55:49.325 INFO:teuthology.orchestra.run.smithi083.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T15:55:49.325 INFO:teuthology.orchestra.run.smithi083.stdout:Group : Unspecified 2024-03-28T15:55:49.325 INFO:teuthology.orchestra.run.smithi083.stdout:Size : 8264254 2024-03-28T15:55:49.325 INFO:teuthology.orchestra.run.smithi083.stdout:License : GPLv3+ 2024-03-28T15:55:49.325 INFO:teuthology.orchestra.run.smithi083.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T15:55:49.325 INFO:teuthology.orchestra.run.smithi083.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T15:55:49.325 INFO:teuthology.orchestra.run.smithi083.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T15:55:49.325 INFO:teuthology.orchestra.run.smithi083.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T15:55:49.326 INFO:teuthology.orchestra.run.smithi083.stdout:Packager : builder@centos.org 2024-03-28T15:55:49.326 INFO:teuthology.orchestra.run.smithi083.stdout:Vendor : CentOS 2024-03-28T15:55:49.326 INFO:teuthology.orchestra.run.smithi083.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T15:55:49.326 INFO:teuthology.orchestra.run.smithi083.stdout:Summary : Support tools for GRUB. 2024-03-28T15:55:49.326 INFO:teuthology.orchestra.run.smithi083.stdout:Description : 2024-03-28T15:55:49.326 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T15:55:49.326 INFO:teuthology.orchestra.run.smithi083.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T15:55:49.326 INFO:teuthology.orchestra.run.smithi083.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T15:55:49.326 INFO:teuthology.orchestra.run.smithi083.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T15:55:49.326 INFO:teuthology.orchestra.run.smithi083.stdout:hardware devices. 2024-03-28T15:55:49.326 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T15:55:49.326 INFO:teuthology.orchestra.run.smithi083.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T15:55:49.327 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T15:55:49.327 INFO:teuthology.task.kernel:Updating grub on smithi083 to boot 5.14.0-432.el9.x86_64 2024-03-28T15:55:49.327 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T15:55:49.953 INFO:teuthology.orchestra.run.smithi083.stderr:Generating grub configuration file ... 2024-03-28T15:55:51.408 INFO:teuthology.orchestra.run.smithi083.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T15:55:51.443 INFO:teuthology.orchestra.run.smithi083.stderr:done 2024-03-28T15:55:51.445 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2024-03-28T15:55:51.460 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.VrhKChpz22 2024-03-28T15:55:51.460 DEBUG:teuthology.orchestra.run.smithi083:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.VrhKChpz22 2024-03-28T15:55:51.528 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.VrhKChpz22 2024-03-28T15:55:51.667 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.VrhKChpz22 is 6KB 2024-03-28T15:55:51.717 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.VrhKChpz22 2024-03-28T15:55:51.731 DEBUG:teuthology.orchestra.run.smithi083:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T15:55:51.798 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T15:55:51.798 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T15:55:51.798 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T15:55:51.798 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-28T15:55:51.799 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-28T15:55:51.967 DEBUG:teuthology.orchestra.run.smithi083:> sudo shutdown -r now 2024-03-28T15:55:52.036 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T15:55:52.037 INFO:teuthology.orchestra.run.smithi007.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T15:55:52.037 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T15:55:52.083 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-28T15:55:52.523 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-28T15:55:52.523 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-28T15:55:52.523 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T15:55:52.523 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T15:55:52.939 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T15:55:52.940 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T15:55:52.940 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T15:55:52.940 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-28T15:55:52.940 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-28T15:55:52.940 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-28T15:55:52.940 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-28T15:55:52.940 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T15:55:52.940 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T15:55:53.258 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T15:55:53.417 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-28T15:55:53.417 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-28T15:55:53.417 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-28T15:55:53.417 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-28T15:55:53.417 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-28T15:55:53.418 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-28T15:55:53.418 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=None, version=distro) 2024-03-28T15:55:53.418 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T15:55:53.501 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-28T15:55:53.501 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-28T15:55:53.501 DEBUG:teuthology.orchestra.run.smithi007:> sudo rpm -qi grub2-tools 2024-03-28T15:55:53.544 INFO:teuthology.orchestra.run.smithi007.stdout:Name : grub2-tools 2024-03-28T15:55:53.544 INFO:teuthology.orchestra.run.smithi007.stdout:Epoch : 1 2024-03-28T15:55:53.544 INFO:teuthology.orchestra.run.smithi007.stdout:Version : 2.06 2024-03-28T15:55:53.544 INFO:teuthology.orchestra.run.smithi007.stdout:Release : 68.el9 2024-03-28T15:55:53.544 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: x86_64 2024-03-28T15:55:53.544 INFO:teuthology.orchestra.run.smithi007.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T15:55:53.544 INFO:teuthology.orchestra.run.smithi007.stdout:Group : Unspecified 2024-03-28T15:55:53.544 INFO:teuthology.orchestra.run.smithi007.stdout:Size : 8264254 2024-03-28T15:55:53.544 INFO:teuthology.orchestra.run.smithi007.stdout:License : GPLv3+ 2024-03-28T15:55:53.544 INFO:teuthology.orchestra.run.smithi007.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T15:55:53.544 INFO:teuthology.orchestra.run.smithi007.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T15:55:53.544 INFO:teuthology.orchestra.run.smithi007.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T15:55:53.545 INFO:teuthology.orchestra.run.smithi007.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T15:55:53.545 INFO:teuthology.orchestra.run.smithi007.stdout:Packager : builder@centos.org 2024-03-28T15:55:53.545 INFO:teuthology.orchestra.run.smithi007.stdout:Vendor : CentOS 2024-03-28T15:55:53.545 INFO:teuthology.orchestra.run.smithi007.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T15:55:53.545 INFO:teuthology.orchestra.run.smithi007.stdout:Summary : Support tools for GRUB. 2024-03-28T15:55:53.545 INFO:teuthology.orchestra.run.smithi007.stdout:Description : 2024-03-28T15:55:53.545 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T15:55:53.545 INFO:teuthology.orchestra.run.smithi007.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T15:55:53.545 INFO:teuthology.orchestra.run.smithi007.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T15:55:53.545 INFO:teuthology.orchestra.run.smithi007.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T15:55:53.545 INFO:teuthology.orchestra.run.smithi007.stdout:hardware devices. 2024-03-28T15:55:53.545 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T15:55:53.545 INFO:teuthology.orchestra.run.smithi007.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T15:55:53.546 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T15:55:53.546 INFO:teuthology.task.kernel:Updating grub on smithi007 to boot 5.14.0-432.el9.x86_64 2024-03-28T15:55:53.546 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T15:55:54.219 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2024-03-28T15:55:55.588 INFO:teuthology.orchestra.run.smithi007.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T15:55:55.637 INFO:teuthology.orchestra.run.smithi007.stderr:done 2024-03-28T15:55:55.639 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-03-28T15:55:55.655 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.mJi1kLOgCe 2024-03-28T15:55:55.655 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mJi1kLOgCe 2024-03-28T15:55:55.727 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.mJi1kLOgCe 2024-03-28T15:55:55.877 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.mJi1kLOgCe is 6KB 2024-03-28T15:55:55.925 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.mJi1kLOgCe 2024-03-28T15:55:55.940 DEBUG:teuthology.orchestra.run.smithi007:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T15:55:56.009 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T15:55:56.009 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T15:55:56.009 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T15:55:56.009 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-28T15:55:56.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-28T15:55:56.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2024-03-28T15:56:21.996 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-28T15:56:21.997 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-28T15:56:21.998 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:56:26.162 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-28T15:56:26.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-28T15:56:26.163 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:57:22.056 DEBUG:teuthology.orchestra.remote:timed out 2024-03-28T15:57:26.169 DEBUG:teuthology.orchestra.remote:timed out 2024-03-28T15:57:31.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-28T15:57:31.059 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:57:32.454 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-03-28T15:57:33.104 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-28T15:57:33.104 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-28T15:57:33.105 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-03-28T15:57:33.119 INFO:teuthology.orchestra.run.smithi083.stdout:5.14.0-432.el9.x86_64 2024-03-28T15:57:33.119 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-28T15:57:33.119 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T15:57:33.119 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T15:57:34.121 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-28T15:57:34.121 DEBUG:teuthology.orchestra.run.smithi083:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-28T15:57:34.216 INFO:teuthology.orchestra.run.smithi083.stdout:ttyS1 2024-03-28T15:57:34.257 DEBUG:teuthology.parallel:result is None 2024-03-28T15:57:35.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-28T15:57:35.170 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:57:35.513 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-03-28T15:57:35.866 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-28T15:57:35.866 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-28T15:57:35.866 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-03-28T15:57:35.880 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-432.el9.x86_64 2024-03-28T15:57:35.880 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-28T15:57:35.880 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T15:57:35.880 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T15:57:36.881 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-28T15:57:36.881 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-28T15:57:36.962 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2024-03-28T15:57:36.994 DEBUG:teuthology.parallel:result is None 2024-03-28T15:57:36.995 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-28T15:57:37.007 INFO:teuthology.task.internal:Creating test directory... 2024-03-28T15:57:37.007 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T15:57:37.010 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T15:57:37.025 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-28T15:57:37.047 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-28T15:57:37.079 INFO:teuthology.task.internal:Creating archive directory... 2024-03-28T15:57:37.080 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T15:57:37.082 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T15:57:37.111 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-28T15:57:37.119 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-28T15:57:37.120 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T15:57:37.153 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T15:57:37.177 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T15:57:37.190 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T15:57:37.223 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T15:57:37.235 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T15:57:37.237 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-28T15:57:37.246 INFO:teuthology.task.internal:Configuring sudo... 2024-03-28T15:57:37.246 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T15:57:37.248 DEBUG:teuthology.orchestra.run.smithi083:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T15:57:37.274 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-28T15:57:37.288 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-28T15:57:37.288 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T15:57:37.318 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T15:57:37.333 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T15:57:37.415 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T15:57:37.473 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T15:57:37.554 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T15:57:37.605 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T15:57:37.605 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T15:57:37.675 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T15:57:37.716 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T15:57:37.761 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T15:57:37.838 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T15:57:37.876 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-28T15:57:37.877 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T15:57:37.941 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2024-03-28T15:57:37.943 DEBUG:teuthology.orchestra.run.smithi083:> sudo service rsyslog restart 2024-03-28T15:57:38.009 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T15:57:38.052 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T15:57:38.524 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-28T15:57:38.534 INFO:teuthology.task.internal:Starting timer... 2024-03-28T15:57:38.535 INFO:teuthology.run_tasks:Running task pcp... 2024-03-28T15:57:38.567 INFO:teuthology.run_tasks:Running task selinux... 2024-03-28T15:57:38.598 DEBUG:teuthology.orchestra.run.smithi007:> sudo service auditd rotate 2024-03-28T15:57:38.694 INFO:teuthology.orchestra.run.smithi007.stdout:Rotating logs: 2024-03-28T15:57:38.697 DEBUG:teuthology.orchestra.run.smithi083:> sudo service auditd rotate 2024-03-28T15:57:38.876 INFO:teuthology.orchestra.run.smithi083.stdout:Rotating logs: 2024-03-28T15:57:38.878 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-28T15:57:38.878 DEBUG:teuthology.orchestra.run.smithi007:> /usr/sbin/getenforce 2024-03-28T15:57:38.915 INFO:teuthology.orchestra.run.smithi007.stdout:Permissive 2024-03-28T15:57:38.915 DEBUG:teuthology.orchestra.run.smithi083:> /usr/sbin/getenforce 2024-03-28T15:57:38.933 INFO:teuthology.orchestra.run.smithi083.stdout:Permissive 2024-03-28T15:57:38.934 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi007.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi083.front.sepia.ceph.com': 'permissive'} 2024-03-28T15:57:38.934 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-28T15:57:38.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T15:57:38.996 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-28T15:57:39.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T15:57:39.024 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-28T15:57:39.024 DEBUG:teuthology.orchestra.run.smithi007:> sudo /usr/sbin/setenforce permissive 2024-03-28T15:57:39.074 DEBUG:teuthology.orchestra.run.smithi083:> sudo /usr/sbin/setenforce permissive 2024-03-28T15:57:39.369 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-28T15:57:39.379 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-28T15:57:39.396 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-28T15:57:39.397 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi083.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-28T16:04:18.250 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi083.front.sepia.ceph.com')] 2024-03-28T16:04:18.280 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-28T16:04:18.281 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T16:04:18.370 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-03-28T16:04:18.449 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-28T16:04:18.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-28T16:04:18.450 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T16:04:18.538 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-03-28T16:04:18.613 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-28T16:04:18.613 INFO:teuthology.run_tasks:Running task clock... 2024-03-28T16:04:18.624 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-28T16:04:18.625 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T16:04:18.625 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T16:04:18.627 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T16:04:18.627 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T16:04:18.656 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T16:04:18.673 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T16:04:18.685 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T16:04:18.702 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T16:04:18.703 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: ntpd: command not found 2024-03-28T16:04:18.716 INFO:teuthology.orchestra.run.smithi007.stdout:506 Cannot talk to daemon 2024-03-28T16:04:18.731 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: ntpd: command not found 2024-03-28T16:04:18.732 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T16:04:18.745 INFO:teuthology.orchestra.run.smithi083.stdout:506 Cannot talk to daemon 2024-03-28T16:04:18.749 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T16:04:18.761 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T16:04:18.777 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T16:04:18.806 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2024-03-28T16:04:18.808 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T16:04:18.808 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-03-28T16:04:18.809 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T16:04:18.809 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T16:04:18.809 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T16:04:18.809 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T16:04:18.831 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found 2024-03-28T16:04:18.834 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T16:04:18.835 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-03-28T16:04:18.835 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T16:04:18.835 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T16:04:18.835 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T16:04:18.835 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T16:04:18.836 INFO:teuthology.run_tasks:Running task install... 2024-03-28T16:04:18.844 DEBUG:teuthology.task.install:project ceph 2024-03-28T16:04:18.844 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'e58b9979847f029f7ab1514b5f5f99b9f03d97f6'}} 2024-03-28T16:04:18.844 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': 'e58b9979847f029f7ab1514b5f5f99b9f03d97f6'} 2024-03-28T16:04:18.844 INFO:teuthology.task.install:Using flavor: default 2024-03-28T16:04:18.850 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-28T16:04:18.850 INFO:teuthology.task.install:extra packages: [] 2024-03-28T16:04:18.851 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e58b9979847f029f7ab1514b5f5f99b9f03d97f6', 'tag': None, 'wait_for_package': False} 2024-03-28T16:04:18.851 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e58b9979847f029f7ab1514b5f5f99b9f03d97f6 2024-03-28T16:04:18.853 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e58b9979847f029f7ab1514b5f5f99b9f03d97f6', 'tag': None, 'wait_for_package': False} 2024-03-28T16:04:18.853 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e58b9979847f029f7ab1514b5f5f99b9f03d97f6 2024-03-28T16:04:18.998 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-03-26-1132-reef/e58b9979847f029f7ab1514b5f5f99b9f03d97f6/centos/9/flavors/default/ 2024-03-28T16:04:18.998 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1053.ge58b9979 2024-03-28T16:04:19.002 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-03-26-1132-reef/e58b9979847f029f7ab1514b5f5f99b9f03d97f6/centos/9/flavors/default/ 2024-03-28T16:04:19.002 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1053.ge58b9979 2024-03-28T16:04:19.115 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-03-26-1132-reef/e58b9979847f029f7ab1514b5f5f99b9f03d97f6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-03-26-1132-reef/e58b9979847f029f7ab1514b5f5f99b9f03d97f6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-03-26-1132-reef/e58b9979847f029f7ab1514b5f5f99b9f03d97f6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T16:04:19.116 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-28T16:04:19.116 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T16:04:19.144 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-28T16:04:19.144 DEBUG:teuthology.orchestra.run.smithi083:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e58b9979847f029f7ab1514b5f5f99b9f03d97f6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T16:04:19.169 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-03-26-1132-reef/e58b9979847f029f7ab1514b5f5f99b9f03d97f6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-03-26-1132-reef/e58b9979847f029f7ab1514b5f5f99b9f03d97f6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-03-26-1132-reef/e58b9979847f029f7ab1514b5f5f99b9f03d97f6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T16:04:19.169 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:04:19.170 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T16:04:19.198 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-28T16:04:19.198 DEBUG:teuthology.orchestra.run.smithi007:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e58b9979847f029f7ab1514b5f5f99b9f03d97f6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T16:04:19.211 DEBUG:teuthology.orchestra.run.smithi083:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T16:04:19.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T16:04:19.290 DEBUG:teuthology.orchestra.run.smithi083:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T16:04:19.345 DEBUG:teuthology.orchestra.run.smithi007:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T16:04:19.357 INFO:teuthology.orchestra.run.smithi083.stdout:check_obsoletes = 1 2024-03-28T16:04:19.359 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum clean all 2024-03-28T16:04:19.412 INFO:teuthology.orchestra.run.smithi007.stdout:check_obsoletes = 1 2024-03-28T16:04:19.413 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum clean all 2024-03-28T16:04:19.623 INFO:teuthology.orchestra.run.smithi083.stdout:74 files removed 2024-03-28T16:04:19.650 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-radosgw 2024-03-28T16:04:19.661 INFO:teuthology.orchestra.run.smithi007.stdout:74 files removed 2024-03-28T16:04:19.688 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-radosgw 2024-03-28T16:04:20.389 INFO:teuthology.orchestra.run.smithi007.stdout:ceph packages for x86_64 194 kB/s | 77 kB 00:00 2024-03-28T16:04:20.408 INFO:teuthology.orchestra.run.smithi083.stdout:ceph packages for x86_64 166 kB/s | 77 kB 00:00 2024-03-28T16:04:20.798 INFO:teuthology.orchestra.run.smithi007.stdout:ceph noarch packages 28 kB/s | 9.9 kB 00:00 2024-03-28T16:04:20.934 INFO:teuthology.orchestra.run.smithi083.stdout:ceph noarch packages 21 kB/s | 9.9 kB 00:00 2024-03-28T16:04:21.099 INFO:teuthology.orchestra.run.smithi007.stdout:ceph source packages 7.2 kB/s | 1.9 kB 00:00 2024-03-28T16:04:21.526 INFO:teuthology.orchestra.run.smithi083.stdout:ceph source packages 3.4 kB/s | 1.9 kB 00:00 2024-03-28T16:04:22.209 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - AppStream 6.2 MB/s | 6.7 MB 00:01 2024-03-28T16:04:22.662 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - AppStream 6.1 MB/s | 6.7 MB 00:01 2024-03-28T16:04:23.676 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2024-03-28T16:04:24.112 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2024-03-28T16:04:25.070 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-28T16:04:27.812 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-28T16:04:31.822 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-28T16:04:33.348 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - Extras packages 90 kB/s | 16 kB 00:00 2024-03-28T16:04:33.524 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 867 kB/s | 8.0 MB 00:09 2024-03-28T16:04:35.743 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 9.0 MB/s | 21 MB 00:02 2024-03-28T16:04:38.833 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 4.5 MB/s | 19 MB 00:04 2024-03-28T16:04:40.616 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-28T16:04:42.958 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-28T16:04:43.050 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:04:43.068 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:04:43.068 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-28T16:04:43.068 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:04:43.069 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:04:43.069 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 7.7 M 2024-03-28T16:04:43.069 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrading: 2024-03-28T16:04:43.069 INFO:teuthology.orchestra.run.smithi083.stdout: librados2 x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 3.3 M 2024-03-28T16:04:43.069 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1 x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 3.0 M 2024-03-28T16:04:43.069 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-28T16:04:43.069 INFO:teuthology.orchestra.run.smithi083.stdout: boost-program-options 2024-03-28T16:04:43.069 INFO:teuthology.orchestra.run.smithi083.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T16:04:43.069 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 5.1 M 2024-03-28T16:04:43.069 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 18 M 2024-03-28T16:04:43.069 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 25 k 2024-03-28T16:04:43.069 INFO:teuthology.orchestra.run.smithi083.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-28T16:04:43.069 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T16:04:43.069 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T16:04:43.070 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T16:04:43.070 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2 x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 709 k 2024-03-28T16:04:43.070 INFO:teuthology.orchestra.run.smithi083.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T16:04:43.070 INFO:teuthology.orchestra.run.smithi083.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T16:04:43.070 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T16:04:43.070 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1 x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 474 k 2024-03-28T16:04:43.070 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T16:04:43.070 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2 x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 4.5 M 2024-03-28T16:04:43.070 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T16:04:43.070 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T16:04:43.070 INFO:teuthology.orchestra.run.smithi083.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T16:04:43.070 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse 2024-03-28T16:04:43.070 INFO:teuthology.orchestra.run.smithi083.stdout: x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 45 k 2024-03-28T16:04:43.070 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 129 k 2024-03-28T16:04:43.071 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 162 k 2024-03-28T16:04:43.071 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T16:04:43.071 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 321 k 2024-03-28T16:04:43.071 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 298 k 2024-03-28T16:04:43.071 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 99 k 2024-03-28T16:04:43.071 INFO:teuthology.orchestra.run.smithi083.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T16:04:43.071 INFO:teuthology.orchestra.run.smithi083.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T16:04:43.071 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:04:43.071 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:04:43.071 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:04:43.071 INFO:teuthology.orchestra.run.smithi083.stdout:Install 28 Packages 2024-03-28T16:04:43.071 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrade 2 Packages 2024-03-28T16:04:43.071 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:04:43.072 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 53 M 2024-03-28T16:04:43.072 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:04:44.426 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 16 kB 00:00 2024-03-28T16:04:44.584 INFO:teuthology.orchestra.run.smithi083.stdout:(1/30): ceph-common-18.2.2-1053.ge58b9979.el9.x 14 MB/s | 18 MB 00:01 2024-03-28T16:04:44.626 INFO:teuthology.orchestra.run.smithi083.stdout:(2/30): ceph-selinux-18.2.2-1053.ge58b9979.el9. 607 kB/s | 25 kB 00:00 2024-03-28T16:04:44.709 INFO:teuthology.orchestra.run.smithi083.stdout:(3/30): libcephfs2-18.2.2-1053.ge58b9979.el9.x8 8.3 MB/s | 709 kB 00:00 2024-03-28T16:04:44.768 INFO:teuthology.orchestra.run.smithi083.stdout:(4/30): libradosstriper1-18.2.2-1053.ge58b9979. 7.9 MB/s | 474 kB 00:00 2024-03-28T16:04:45.051 INFO:teuthology.orchestra.run.smithi083.stdout:(5/30): librgw2-18.2.2-1053.ge58b9979.el9.x86_6 16 MB/s | 4.5 MB 00:00 2024-03-28T16:04:45.101 INFO:teuthology.orchestra.run.smithi083.stdout:(6/30): python3-ceph-argparse-18.2.2-1053.ge58b 900 kB/s | 45 kB 00:00 2024-03-28T16:04:45.160 INFO:teuthology.orchestra.run.smithi083.stdout:(7/30): python3-ceph-common-18.2.2-1053.ge58b99 2.2 MB/s | 129 kB 00:00 2024-03-28T16:04:45.219 INFO:teuthology.orchestra.run.smithi083.stdout:(8/30): python3-cephfs-18.2.2-1053.ge58b9979.el 2.7 MB/s | 162 kB 00:00 2024-03-28T16:04:45.294 INFO:teuthology.orchestra.run.smithi083.stdout:(9/30): ceph-base-18.2.2-1053.ge58b9979.el9.x86 2.5 MB/s | 5.1 MB 00:02 2024-03-28T16:04:45.320 INFO:teuthology.orchestra.run.smithi083.stdout:(10/30): python3-rados-18.2.2-1053.ge58b9979.el 3.1 MB/s | 321 kB 00:00 2024-03-28T16:04:45.361 INFO:teuthology.orchestra.run.smithi083.stdout:(11/30): python3-rbd-18.2.2-1053.ge58b9979.el9. 4.4 MB/s | 298 kB 00:00 2024-03-28T16:04:45.387 INFO:teuthology.orchestra.run.smithi083.stdout:(12/30): python3-rgw-18.2.2-1053.ge58b9979.el9. 1.5 MB/s | 99 kB 00:00 2024-03-28T16:04:45.645 INFO:teuthology.orchestra.run.smithi083.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 373 kB/s | 106 kB 00:00 2024-03-28T16:04:45.696 INFO:teuthology.orchestra.run.smithi083.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 516 kB/s | 159 kB 00:00 2024-03-28T16:04:45.721 INFO:teuthology.orchestra.run.smithi083.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 589 kB/s | 44 kB 00:00 2024-03-28T16:04:45.963 INFO:teuthology.orchestra.run.smithi083.stdout:(16/30): ceph-radosgw-18.2.2-1053.ge58b9979.el9 2.9 MB/s | 7.7 MB 00:02 2024-03-28T16:04:45.997 INFO:teuthology.orchestra.run.smithi083.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 2.1 MB/s | 661 kB 00:00 2024-03-28T16:04:46.030 INFO:teuthology.orchestra.run.smithi083.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 915 kB/s | 282 kB 00:00 2024-03-28T16:04:46.055 INFO:teuthology.orchestra.run.smithi083.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 453 kB/s | 41 kB 00:00 2024-03-28T16:04:46.436 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-28T16:04:46.505 INFO:teuthology.orchestra.run.smithi083.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 233 kB/s | 111 kB 00:00 2024-03-28T16:04:46.622 INFO:teuthology.orchestra.run.smithi083.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 522 kB/s | 309 kB 00:00 2024-03-28T16:04:46.648 INFO:teuthology.orchestra.run.smithi083.stdout:(22/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 183 kB/s | 26 kB 00:00 2024-03-28T16:04:46.756 INFO:teuthology.orchestra.run.smithi083.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 364 kB/s | 49 kB 00:00 2024-03-28T16:04:46.781 INFO:teuthology.orchestra.run.smithi083.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 504 kB/s | 67 kB 00:00 2024-03-28T16:04:46.924 INFO:teuthology.orchestra.run.smithi083.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-28T16:04:47.474 INFO:teuthology.orchestra.run.smithi083.stdout:(26/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.1 MB/s | 838 kB 00:00 2024-03-28T16:04:47.549 INFO:teuthology.orchestra.run.smithi083.stdout:(27/30): libarrow-9.0.0-10.el9.x86_64.rpm 3.0 MB/s | 4.4 MB 00:01 2024-03-28T16:04:47.674 INFO:teuthology.orchestra.run.smithi083.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 2.1 MB/s | 1.6 MB 00:00 2024-03-28T16:04:47.874 INFO:teuthology.orchestra.run.smithi083.stdout:(29/30): librbd1-18.2.2-1053.ge58b9979.el9.x86_ 9.3 MB/s | 3.0 MB 00:00 2024-03-28T16:04:48.274 INFO:teuthology.orchestra.run.smithi083.stdout:(30/30): librados2-18.2.2-1053.ge58b9979.el9.x8 4.1 MB/s | 3.3 MB 00:00 2024-03-28T16:04:48.279 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:04:48.279 INFO:teuthology.orchestra.run.smithi083.stdout:Total 10 MB/s | 53 MB 00:05 2024-03-28T16:04:48.279 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:04:48.315 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:04:48.315 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:04:48.745 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:04:48.745 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:04:50.128 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:04:50.215 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-03-28T16:04:50.246 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-03-28T16:04:50.296 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-03-28T16:04:50.619 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-03-28T16:04:50.679 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librados2-2:18.2.2-1053.ge58b9979.el9.x86_64 5/32 2024-03-28T16:04:50.770 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-2:18.2.2-1053.ge58b9979.el9.x86_64 5/32 2024-03-28T16:04:50.854 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rados-2:18.2.2-1053.ge58b9979.el9.x86_64 6/32 2024-03-28T16:04:50.880 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs2-2:18.2.2-1053.ge58b9979.el9.x86_64 7/32 2024-03-28T16:04:50.934 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs2-2:18.2.2-1053.ge58b9979.el9.x86_64 7/32 2024-03-28T16:04:50.963 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-03-28T16:04:51.031 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-argparse-2:18.2.2-1053.ge58b9979.el9. 9/32 2024-03-28T16:04:51.097 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cephfs-2:18.2.2-1053.ge58b9979.el9.x86_64 10/32 2024-03-28T16:04:51.124 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libradosstriper1-2:18.2.2-1053.ge58b9979.el9.x86_6 11/32 2024-03-28T16:04:51.173 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1053.ge58b9979.el9.x86_6 11/32 2024-03-28T16:04:51.278 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-03-28T16:04:51.316 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-03-28T16:04:51.355 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-03-28T16:04:51.396 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-03-28T16:04:51.423 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-03-28T16:04:51.459 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-03-28T16:04:51.492 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-03-28T16:04:51.510 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-03-28T16:04:51.802 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-03-28T16:04:51.830 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librbd1-2:18.2.2-1053.ge58b9979.el9.x86_64 20/32 2024-03-28T16:04:51.912 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-2:18.2.2-1053.ge58b9979.el9.x86_64 20/32 2024-03-28T16:04:51.943 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rbd-2:18.2.2-1053.ge58b9979.el9.x86_64 21/32 2024-03-28T16:04:51.989 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-03-28T16:04:52.387 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-03-28T16:04:52.416 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librgw2-2:18.2.2-1053.ge58b9979.el9.x86_64 24/32 2024-03-28T16:04:52.470 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librgw2-2:18.2.2-1053.ge58b9979.el9.x86_64 24/32 2024-03-28T16:04:52.538 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rgw-2:18.2.2-1053.ge58b9979.el9.x86_64 25/32 2024-03-28T16:04:52.568 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-common-2:18.2.2-1053.ge58b9979.el9.x8 26/32 2024-03-28T16:04:54.076 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:04:54.095 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:04:54.095 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-28T16:04:54.095 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:04:54.096 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:04:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 7.7 M 2024-03-28T16:04:54.096 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrading: 2024-03-28T16:04:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: librados2 x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 3.3 M 2024-03-28T16:04:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1 x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 3.0 M 2024-03-28T16:04:54.096 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-28T16:04:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: boost-program-options 2024-03-28T16:04:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T16:04:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 5.1 M 2024-03-28T16:04:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 18 M 2024-03-28T16:04:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 25 k 2024-03-28T16:04:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-28T16:04:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T16:04:54.096 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T16:04:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T16:04:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2 x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 709 k 2024-03-28T16:04:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T16:04:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T16:04:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T16:04:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1 x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 474 k 2024-03-28T16:04:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T16:04:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2 x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 4.5 M 2024-03-28T16:04:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T16:04:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T16:04:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T16:04:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse 2024-03-28T16:04:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 45 k 2024-03-28T16:04:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 129 k 2024-03-28T16:04:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 162 k 2024-03-28T16:04:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T16:04:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 321 k 2024-03-28T16:04:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 298 k 2024-03-28T16:04:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 99 k 2024-03-28T16:04:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T16:04:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T16:04:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:04:54.098 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:04:54.098 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:04:54.098 INFO:teuthology.orchestra.run.smithi007.stdout:Install 28 Packages 2024-03-28T16:04:54.098 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrade 2 Packages 2024-03-28T16:04:54.098 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:04:54.099 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 53 M 2024-03-28T16:04:54.099 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:04:54.609 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:18.2.2-1053.ge58b9979.el9.x86_64 27/32 2024-03-28T16:04:54.657 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-common-2:18.2.2-1053.ge58b9979.el9.x86_64 27/32 2024-03-28T16:04:55.202 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:18.2.2-1053.ge58b9979.el9.x86_64 27/32 2024-03-28T16:04:55.244 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-base-2:18.2.2-1053.ge58b9979.el9.x86_64 28/32 2024-03-28T16:04:55.326 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-base-2:18.2.2-1053.ge58b9979.el9.x86_64 28/32 2024-03-28T16:04:55.326 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T16:04:55.326 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T16:04:55.327 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:04:55.361 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-selinux-2:18.2.2-1053.ge58b9979.el9.x86_64 29/32 2024-03-28T16:04:56.520 INFO:teuthology.orchestra.run.smithi007.stdout:(1/30): ceph-common-18.2.2-1053.ge58b9979.el9.x 8.2 MB/s | 18 MB 00:02 2024-03-28T16:04:56.570 INFO:teuthology.orchestra.run.smithi007.stdout:(2/30): ceph-selinux-18.2.2-1053.ge58b9979.el9. 505 kB/s | 25 kB 00:00 2024-03-28T16:04:56.679 INFO:teuthology.orchestra.run.smithi007.stdout:(3/30): libcephfs2-18.2.2-1053.ge58b9979.el9.x8 6.4 MB/s | 709 kB 00:00 2024-03-28T16:04:56.746 INFO:teuthology.orchestra.run.smithi007.stdout:(4/30): libradosstriper1-18.2.2-1053.ge58b9979. 6.9 MB/s | 474 kB 00:00 2024-03-28T16:04:56.863 INFO:teuthology.orchestra.run.smithi007.stdout:(5/30): ceph-base-18.2.2-1053.ge58b9979.el9.x86 2.0 MB/s | 5.1 MB 00:02 2024-03-28T16:04:56.971 INFO:teuthology.orchestra.run.smithi007.stdout:(6/30): python3-ceph-argparse-18.2.2-1053.ge58b 415 kB/s | 45 kB 00:00 2024-03-28T16:04:57.097 INFO:teuthology.orchestra.run.smithi007.stdout:(7/30): python3-ceph-common-18.2.2-1053.ge58b99 1.0 MB/s | 129 kB 00:00 2024-03-28T16:04:57.205 INFO:teuthology.orchestra.run.smithi007.stdout:(8/30): librgw2-18.2.2-1053.ge58b9979.el9.x86_6 9.7 MB/s | 4.5 MB 00:00 2024-03-28T16:04:57.230 INFO:teuthology.orchestra.run.smithi007.stdout:(9/30): python3-cephfs-18.2.2-1053.ge58b9979.el 1.2 MB/s | 162 kB 00:00 2024-03-28T16:04:57.264 INFO:teuthology.orchestra.run.smithi007.stdout:(10/30): python3-rados-18.2.2-1053.ge58b9979.el 5.4 MB/s | 321 kB 00:00 2024-03-28T16:04:57.331 INFO:teuthology.orchestra.run.smithi007.stdout:(11/30): python3-rgw-18.2.2-1053.ge58b9979.el9. 1.5 MB/s | 99 kB 00:00 2024-03-28T16:04:57.365 INFO:teuthology.orchestra.run.smithi007.stdout:(12/30): python3-rbd-18.2.2-1053.ge58b9979.el9. 2.2 MB/s | 298 kB 00:00 2024-03-28T16:04:57.598 INFO:teuthology.orchestra.run.smithi007.stdout:(13/30): ceph-radosgw-18.2.2-1053.ge58b9979.el9 2.3 MB/s | 7.7 MB 00:03 2024-03-28T16:04:57.623 INFO:teuthology.orchestra.run.smithi007.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 362 kB/s | 106 kB 00:00 2024-03-28T16:04:57.666 INFO:teuthology.orchestra.run.smithi007.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 663 kB/s | 44 kB 00:00 2024-03-28T16:04:57.766 INFO:teuthology.orchestra.run.smithi007.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 397 kB/s | 159 kB 00:00 2024-03-28T16:04:57.833 INFO:teuthology.orchestra.run.smithi007.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 661 kB 00:00 2024-03-28T16:04:57.858 INFO:teuthology.orchestra.run.smithi007.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 282 kB 00:00 2024-03-28T16:04:57.875 INFO:teuthology.orchestra.run.smithi007.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 379 kB/s | 41 kB 00:00 2024-03-28T16:04:58.417 INFO:teuthology.orchestra.run.smithi007.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 190 kB/s | 111 kB 00:00 2024-03-28T16:04:58.525 INFO:teuthology.orchestra.run.smithi007.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 463 kB/s | 309 kB 00:00 2024-03-28T16:04:58.542 INFO:teuthology.orchestra.run.smithi007.stdout:(22/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 208 kB/s | 26 kB 00:00 2024-03-28T16:04:58.626 INFO:teuthology.orchestra.run.smithi007.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2024-03-28T16:04:58.684 INFO:teuthology.orchestra.run.smithi007.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 474 kB/s | 67 kB 00:00 2024-03-28T16:04:58.876 INFO:teuthology.orchestra.run.smithi007.stdout:(25/30): libarrow-9.0.0-10.el9.x86_64.rpm 4.4 MB/s | 4.4 MB 00:01 2024-03-28T16:04:58.902 INFO:teuthology.orchestra.run.smithi007.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 884 kB/s | 192 kB 00:00 2024-03-28T16:04:58.927 INFO:teuthology.orchestra.run.smithi007.stdout:(27/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.7 MB/s | 838 kB 00:00 2024-03-28T16:04:59.010 INFO:teuthology.orchestra.run.smithi007.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-03-28T16:04:59.286 INFO:teuthology.orchestra.run.smithi007.stdout:(29/30): librbd1-18.2.2-1053.ge58b9979.el9.x86_ 8.4 MB/s | 3.0 MB 00:00 2024-03-28T16:04:59.736 INFO:teuthology.orchestra.run.smithi007.stdout:(30/30): librados2-18.2.2-1053.ge58b9979.el9.x8 4.0 MB/s | 3.3 MB 00:00 2024-03-28T16:04:59.743 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:04:59.743 INFO:teuthology.orchestra.run.smithi007.stdout:Total 9.4 MB/s | 53 MB 00:05 2024-03-28T16:04:59.744 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:04:59.784 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:04:59.785 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:05:00.251 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:05:00.251 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:05:01.674 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:05:01.761 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-03-28T16:05:01.799 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-03-28T16:05:01.851 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-03-28T16:05:02.254 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-03-28T16:05:02.290 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librados2-2:18.2.2-1053.ge58b9979.el9.x86_64 5/32 2024-03-28T16:05:02.366 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:18.2.2-1053.ge58b9979.el9.x86_64 5/32 2024-03-28T16:05:02.460 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rados-2:18.2.2-1053.ge58b9979.el9.x86_64 6/32 2024-03-28T16:05:02.491 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs2-2:18.2.2-1053.ge58b9979.el9.x86_64 7/32 2024-03-28T16:05:02.553 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs2-2:18.2.2-1053.ge58b9979.el9.x86_64 7/32 2024-03-28T16:05:02.583 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-03-28T16:05:02.627 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-argparse-2:18.2.2-1053.ge58b9979.el9. 9/32 2024-03-28T16:05:02.694 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cephfs-2:18.2.2-1053.ge58b9979.el9.x86_64 10/32 2024-03-28T16:05:02.718 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libradosstriper1-2:18.2.2-1053.ge58b9979.el9.x86_6 11/32 2024-03-28T16:05:02.784 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1053.ge58b9979.el9.x86_6 11/32 2024-03-28T16:05:02.895 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-03-28T16:05:02.927 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-03-28T16:05:02.967 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-03-28T16:05:02.999 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-03-28T16:05:03.034 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-03-28T16:05:03.070 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-03-28T16:05:03.103 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-03-28T16:05:03.425 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-03-28T16:05:03.458 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librbd1-2:18.2.2-1053.ge58b9979.el9.x86_64 20/32 2024-03-28T16:05:03.553 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:18.2.2-1053.ge58b9979.el9.x86_64 20/32 2024-03-28T16:05:03.587 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rbd-2:18.2.2-1053.ge58b9979.el9.x86_64 21/32 2024-03-28T16:05:03.624 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-03-28T16:05:04.049 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-03-28T16:05:04.085 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librgw2-2:18.2.2-1053.ge58b9979.el9.x86_64 24/32 2024-03-28T16:05:04.140 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librgw2-2:18.2.2-1053.ge58b9979.el9.x86_64 24/32 2024-03-28T16:05:04.219 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rgw-2:18.2.2-1053.ge58b9979.el9.x86_64 25/32 2024-03-28T16:05:04.254 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-common-2:18.2.2-1053.ge58b9979.el9.x8 26/32 2024-03-28T16:05:04.876 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1053.ge58b9979.el9.x86_64 29/32 2024-03-28T16:05:04.876 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /sys 2024-03-28T16:05:04.876 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /proc 2024-03-28T16:05:04.876 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /mnt 2024-03-28T16:05:04.876 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /var/tmp 2024-03-28T16:05:04.876 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /home 2024-03-28T16:05:04.876 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /root 2024-03-28T16:05:04.877 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /tmp 2024-03-28T16:05:04.877 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:05.600 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-radosgw-2:18.2.2-1053.ge58b9979.el9.x86_64 30/32 2024-03-28T16:05:05.621 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1053.ge58b9979.el9.x86_64 30/32 2024-03-28T16:05:05.622 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T16:05:05.622 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T16:05:05.622 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T16:05:05.622 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T16:05:05.622 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:05.624 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-28T16:05:05.694 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-28T16:05:05.694 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-28T16:05:06.374 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:18.2.2-1053.ge58b9979.el9.x86_64 27/32 2024-03-28T16:05:06.417 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-common-2:18.2.2-1053.ge58b9979.el9.x86_64 27/32 2024-03-28T16:05:06.694 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-28T16:05:06.694 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-base-2:18.2.2-1053.ge58b9979.el9.x86_64 1/32 2024-03-28T16:05:06.695 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-common-2:18.2.2-1053.ge58b9979.el9.x86_64 2/32 2024-03-28T16:05:06.695 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-radosgw-2:18.2.2-1053.ge58b9979.el9.x86_64 3/32 2024-03-28T16:05:06.695 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-selinux-2:18.2.2-1053.ge58b9979.el9.x86_64 4/32 2024-03-28T16:05:06.695 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs2-2:18.2.2-1053.ge58b9979.el9.x86_64 5/32 2024-03-28T16:05:06.695 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libradosstriper1-2:18.2.2-1053.ge58b9979.el9.x86_6 6/32 2024-03-28T16:05:06.695 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librgw2-2:18.2.2-1053.ge58b9979.el9.x86_64 7/32 2024-03-28T16:05:06.695 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1053.ge58b9979.el9. 8/32 2024-03-28T16:05:06.695 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-common-2:18.2.2-1053.ge58b9979.el9.x8 9/32 2024-03-28T16:05:06.695 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cephfs-2:18.2.2-1053.ge58b9979.el9.x86_64 10/32 2024-03-28T16:05:06.695 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rados-2:18.2.2-1053.ge58b9979.el9.x86_64 11/32 2024-03-28T16:05:06.695 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rbd-2:18.2.2-1053.ge58b9979.el9.x86_64 12/32 2024-03-28T16:05:06.695 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rgw-2:18.2.2-1053.ge58b9979.el9.x86_64 13/32 2024-03-28T16:05:06.695 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-03-28T16:05:06.695 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-03-28T16:05:06.696 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-03-28T16:05:06.696 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-03-28T16:05:06.696 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-03-28T16:05:06.696 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-03-28T16:05:06.696 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-03-28T16:05:06.696 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-03-28T16:05:06.696 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-03-28T16:05:06.696 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-03-28T16:05:06.696 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-03-28T16:05:06.696 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-03-28T16:05:06.696 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-03-28T16:05:06.696 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-03-28T16:05:06.696 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-03-28T16:05:06.696 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-2:18.2.2-1053.ge58b9979.el9.x86_64 29/32 2024-03-28T16:05:06.696 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-03-28T16:05:06.696 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-2:18.2.2-1053.ge58b9979.el9.x86_64 31/32 2024-03-28T16:05:06.983 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:18.2.2-1053.ge58b9979.el9.x86_64 27/32 2024-03-28T16:05:07.021 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-base-2:18.2.2-1053.ge58b9979.el9.x86_64 28/32 2024-03-28T16:05:07.112 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-base-2:18.2.2-1053.ge58b9979.el9.x86_64 28/32 2024-03-28T16:05:07.112 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T16:05:07.112 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T16:05:07.112 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:07.147 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-selinux-2:18.2.2-1053.ge58b9979.el9.x86_64 29/32 2024-03-28T16:05:07.733 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-03-28T16:05:07.733 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:07.734 INFO:teuthology.orchestra.run.smithi083.stdout:Upgraded: 2024-03-28T16:05:07.734 INFO:teuthology.orchestra.run.smithi083.stdout: librados2-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:07.734 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:07.734 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:05:07.734 INFO:teuthology.orchestra.run.smithi083.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T16:05:07.734 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:07.734 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:07.734 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:07.734 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:07.734 INFO:teuthology.orchestra.run.smithi083.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-28T16:05:07.734 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T16:05:07.734 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T16:05:07.734 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T16:05:07.734 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:07.735 INFO:teuthology.orchestra.run.smithi083.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T16:05:07.735 INFO:teuthology.orchestra.run.smithi083.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T16:05:07.735 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T16:05:07.735 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:07.735 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T16:05:07.735 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:07.735 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T16:05:07.735 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T16:05:07.735 INFO:teuthology.orchestra.run.smithi083.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T16:05:07.735 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:07.735 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:07.735 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:07.735 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T16:05:07.735 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:07.735 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:07.736 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:07.736 INFO:teuthology.orchestra.run.smithi083.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T16:05:07.736 INFO:teuthology.orchestra.run.smithi083.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T16:05:07.736 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:07.736 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:05:08.126 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-test 2024-03-28T16:05:08.681 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:28 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:05:08.829 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:05:08.830 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:05:08.830 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-28T16:05:08.830 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:05:08.830 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:05:08.830 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 40 M 2024-03-28T16:05:08.830 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-28T16:05:08.830 INFO:teuthology.orchestra.run.smithi083.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T16:05:08.831 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 166 k 2024-03-28T16:05:08.831 INFO:teuthology.orchestra.run.smithi083.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T16:05:08.831 INFO:teuthology.orchestra.run.smithi083.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T16:05:08.831 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T16:05:08.831 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:08.831 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:05:08.831 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:05:08.831 INFO:teuthology.orchestra.run.smithi083.stdout:Install 6 Packages 2024-03-28T16:05:08.831 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:08.832 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 41 M 2024-03-28T16:05:08.832 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 164 M 2024-03-28T16:05:08.832 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:05:09.071 INFO:teuthology.orchestra.run.smithi083.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-28T16:05:09.171 INFO:teuthology.orchestra.run.smithi083.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.0 MB/s | 300 kB 00:00 2024-03-28T16:05:09.213 INFO:teuthology.orchestra.run.smithi083.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-28T16:05:09.438 INFO:teuthology.orchestra.run.smithi083.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 832 kB/s | 187 kB 00:00 2024-03-28T16:05:09.539 INFO:teuthology.orchestra.run.smithi083.stdout:(5/6): libcephsqlite-18.2.2-1053.ge58b9979.el9. 255 kB/s | 166 kB 00:00 2024-03-28T16:05:15.914 INFO:teuthology.orchestra.run.smithi083.stdout:(6/6): ceph-test-18.2.2-1053.ge58b9979.el9.x86_ 5.7 MB/s | 40 MB 00:07 2024-03-28T16:05:15.917 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:05:15.918 INFO:teuthology.orchestra.run.smithi083.stdout:Total 5.7 MB/s | 41 MB 00:07 2024-03-28T16:05:15.932 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:05:15.952 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:05:15.952 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:05:16.300 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:05:16.301 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:05:16.827 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:05:16.899 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-28T16:05:16.938 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-28T16:05:16.972 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-28T16:05:17.015 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-28T16:05:17.042 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephsqlite-2:18.2.2-1053.ge58b9979.el9.x86_64 5/6 2024-03-28T16:05:17.213 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1053.ge58b9979.el9.x86_64 29/32 2024-03-28T16:05:17.213 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /sys 2024-03-28T16:05:17.213 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /proc 2024-03-28T16:05:17.213 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /mnt 2024-03-28T16:05:17.213 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /var/tmp 2024-03-28T16:05:17.213 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /home 2024-03-28T16:05:17.213 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /root 2024-03-28T16:05:17.213 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /tmp 2024-03-28T16:05:17.213 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:17.984 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-radosgw-2:18.2.2-1053.ge58b9979.el9.x86_64 30/32 2024-03-28T16:05:18.009 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1053.ge58b9979.el9.x86_64 30/32 2024-03-28T16:05:18.009 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T16:05:18.009 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T16:05:18.009 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T16:05:18.009 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T16:05:18.010 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:18.011 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-28T16:05:18.096 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-28T16:05:18.096 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-28T16:05:19.226 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-28T16:05:19.226 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-base-2:18.2.2-1053.ge58b9979.el9.x86_64 1/32 2024-03-28T16:05:19.226 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-common-2:18.2.2-1053.ge58b9979.el9.x86_64 2/32 2024-03-28T16:05:19.226 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-radosgw-2:18.2.2-1053.ge58b9979.el9.x86_64 3/32 2024-03-28T16:05:19.226 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-selinux-2:18.2.2-1053.ge58b9979.el9.x86_64 4/32 2024-03-28T16:05:19.226 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs2-2:18.2.2-1053.ge58b9979.el9.x86_64 5/32 2024-03-28T16:05:19.226 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libradosstriper1-2:18.2.2-1053.ge58b9979.el9.x86_6 6/32 2024-03-28T16:05:19.226 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librgw2-2:18.2.2-1053.ge58b9979.el9.x86_64 7/32 2024-03-28T16:05:19.227 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1053.ge58b9979.el9. 8/32 2024-03-28T16:05:19.227 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-common-2:18.2.2-1053.ge58b9979.el9.x8 9/32 2024-03-28T16:05:19.227 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cephfs-2:18.2.2-1053.ge58b9979.el9.x86_64 10/32 2024-03-28T16:05:19.227 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rados-2:18.2.2-1053.ge58b9979.el9.x86_64 11/32 2024-03-28T16:05:19.227 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rbd-2:18.2.2-1053.ge58b9979.el9.x86_64 12/32 2024-03-28T16:05:19.227 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rgw-2:18.2.2-1053.ge58b9979.el9.x86_64 13/32 2024-03-28T16:05:19.227 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-03-28T16:05:19.227 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-03-28T16:05:19.227 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-03-28T16:05:19.227 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-03-28T16:05:19.227 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-03-28T16:05:19.227 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-03-28T16:05:19.227 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-03-28T16:05:19.227 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-03-28T16:05:19.227 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-03-28T16:05:19.228 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-03-28T16:05:19.228 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-03-28T16:05:19.228 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-03-28T16:05:19.228 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-03-28T16:05:19.228 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-03-28T16:05:19.228 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-03-28T16:05:19.228 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:18.2.2-1053.ge58b9979.el9.x86_64 29/32 2024-03-28T16:05:19.228 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-03-28T16:05:19.228 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:18.2.2-1053.ge58b9979.el9.x86_64 31/32 2024-03-28T16:05:20.355 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-03-28T16:05:20.355 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:20.356 INFO:teuthology.orchestra.run.smithi007.stdout:Upgraded: 2024-03-28T16:05:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:20.356 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:05:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T16:05:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-28T16:05:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T16:05:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T16:05:20.356 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T16:05:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T16:05:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T16:05:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T16:05:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T16:05:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T16:05:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T16:05:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T16:05:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T16:05:20.357 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:20.358 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:20.358 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:20.358 INFO:teuthology.orchestra.run.smithi007.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T16:05:20.358 INFO:teuthology.orchestra.run.smithi007.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T16:05:20.358 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:20.358 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:05:20.691 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-test 2024-03-28T16:05:20.771 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1053.ge58b9979.el9.x86_64 5/6 2024-03-28T16:05:20.841 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-test-2:18.2.2-1053.ge58b9979.el9.x86_64 6/6 2024-03-28T16:05:21.266 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:30 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:05:21.427 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:05:21.427 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:05:21.428 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-28T16:05:21.428 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:05:21.428 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:05:21.428 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 40 M 2024-03-28T16:05:21.428 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-28T16:05:21.428 INFO:teuthology.orchestra.run.smithi007.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T16:05:21.428 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 166 k 2024-03-28T16:05:21.428 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T16:05:21.428 INFO:teuthology.orchestra.run.smithi007.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T16:05:21.428 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T16:05:21.428 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:21.428 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:05:21.428 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:05:21.429 INFO:teuthology.orchestra.run.smithi007.stdout:Install 6 Packages 2024-03-28T16:05:21.429 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:21.429 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 41 M 2024-03-28T16:05:21.429 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 164 M 2024-03-28T16:05:21.429 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:05:21.643 INFO:teuthology.orchestra.run.smithi007.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-28T16:05:21.693 INFO:teuthology.orchestra.run.smithi007.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-28T16:05:21.718 INFO:teuthology.orchestra.run.smithi007.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-28T16:05:21.802 INFO:teuthology.orchestra.run.smithi007.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2024-03-28T16:05:21.852 INFO:teuthology.orchestra.run.smithi007.stdout:(5/6): libcephsqlite-18.2.2-1053.ge58b9979.el9. 455 kB/s | 166 kB 00:00 2024-03-28T16:05:22.927 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-test-2:18.2.2-1053.ge58b9979.el9.x86_64 6/6 2024-03-28T16:05:22.927 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-test-2:18.2.2-1053.ge58b9979.el9.x86_64 1/6 2024-03-28T16:05:22.927 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephsqlite-2:18.2.2-1053.ge58b9979.el9.x86_64 2/6 2024-03-28T16:05:22.927 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-28T16:05:22.927 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-28T16:05:22.927 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-28T16:05:23.296 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-28T16:05:23.296 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:23.297 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:05:23.297 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:23.297 INFO:teuthology.orchestra.run.smithi083.stdout: jq-1.6-16.el9.x86_64 2024-03-28T16:05:23.297 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:23.297 INFO:teuthology.orchestra.run.smithi083.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T16:05:23.297 INFO:teuthology.orchestra.run.smithi083.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T16:05:23.297 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T16:05:23.297 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:23.297 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:05:23.602 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph 2024-03-28T16:05:24.152 INFO:teuthology.orchestra.run.smithi007.stdout:(6/6): ceph-test-18.2.2-1053.ge58b9979.el9.x86_ 15 MB/s | 40 MB 00:02 2024-03-28T16:05:24.155 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:05:24.156 INFO:teuthology.orchestra.run.smithi007.stdout:Total 15 MB/s | 41 MB 00:02 2024-03-28T16:05:24.170 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:05:24.181 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:44 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:05:24.193 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:05:24.193 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:05:24.348 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:05:24.351 INFO:teuthology.orchestra.run.smithi083.stdout:========================================================================================= 2024-03-28T16:05:24.351 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-28T16:05:24.351 INFO:teuthology.orchestra.run.smithi083.stdout:========================================================================================= 2024-03-28T16:05:24.351 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:05:24.351 INFO:teuthology.orchestra.run.smithi083.stdout: ceph x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 6.5 k 2024-03-28T16:05:24.351 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-28T16:05:24.351 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 2.1 M 2024-03-28T16:05:24.351 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 1.5 M 2024-03-28T16:05:24.351 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1053.ge58b9979.el9 ceph-noarch 245 k 2024-03-28T16:05:24.351 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 4.7 M 2024-03-28T16:05:24.351 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 17 M 2024-03-28T16:05:24.351 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T16:05:24.352 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T16:05:24.352 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T16:05:24.352 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T16:05:24.352 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T16:05:24.352 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T16:05:24.352 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T16:05:24.352 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T16:05:24.352 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T16:05:24.352 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T16:05:24.352 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T16:05:24.352 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T16:05:24.352 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T16:05:24.352 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T16:05:24.353 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T16:05:24.353 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T16:05:24.353 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T16:05:24.353 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T16:05:24.353 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T16:05:24.353 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T16:05:24.353 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T16:05:24.353 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T16:05:24.353 INFO:teuthology.orchestra.run.smithi083.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T16:05:24.353 INFO:teuthology.orchestra.run.smithi083.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T16:05:24.353 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T16:05:24.353 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T16:05:24.353 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T16:05:24.353 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:24.353 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:05:24.354 INFO:teuthology.orchestra.run.smithi083.stdout:========================================================================================= 2024-03-28T16:05:24.354 INFO:teuthology.orchestra.run.smithi083.stdout:Install 33 Packages 2024-03-28T16:05:24.354 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:24.354 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 29 M 2024-03-28T16:05:24.354 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 104 M 2024-03-28T16:05:24.354 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:05:24.558 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:05:24.559 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:05:24.842 INFO:teuthology.orchestra.run.smithi083.stdout:(1/33): ceph-18.2.2-1053.ge58b9979.el9.x86_64.r 38 kB/s | 6.5 kB 00:00 2024-03-28T16:05:25.076 INFO:teuthology.orchestra.run.smithi083.stdout:(2/33): ceph-mds-18.2.2-1053.ge58b9979.el9.x86_ 5.2 MB/s | 2.1 MB 00:00 2024-03-28T16:05:25.101 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:05:25.185 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-28T16:05:25.224 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-28T16:05:25.252 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-28T16:05:25.287 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-28T16:05:25.314 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephsqlite-2:18.2.2-1053.ge58b9979.el9.x86_64 5/6 2024-03-28T16:05:25.777 INFO:teuthology.orchestra.run.smithi083.stdout:(3/33): ceph-mgr-18.2.2-1053.ge58b9979.el9.x86_ 1.3 MB/s | 1.5 MB 00:01 2024-03-28T16:05:25.893 INFO:teuthology.orchestra.run.smithi083.stdout:(4/33): ceph-mgr-modules-core-18.2.2-1053.ge58b 2.1 MB/s | 245 kB 00:00 2024-03-28T16:05:26.010 INFO:teuthology.orchestra.run.smithi083.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 144 kB 00:00 2024-03-28T16:05:26.177 INFO:teuthology.orchestra.run.smithi083.stdout:(6/33): ceph-osd-18.2.2-1053.ge58b9979.el9.x86_ 15 MB/s | 17 MB 00:01 2024-03-28T16:05:26.202 INFO:teuthology.orchestra.run.smithi083.stdout:(7/33): python3-markupsafe-1.1.1-12.el9.x86_64. 164 kB/s | 32 kB 00:00 2024-03-28T16:05:26.227 INFO:teuthology.orchestra.run.smithi083.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 951 kB/s | 47 kB 00:00 2024-03-28T16:05:26.252 INFO:teuthology.orchestra.run.smithi083.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 771 kB/s | 39 kB 00:00 2024-03-28T16:05:26.278 INFO:teuthology.orchestra.run.smithi083.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-28T16:05:26.353 INFO:teuthology.orchestra.run.smithi083.stdout:(11/33): ceph-mon-18.2.2-1053.ge58b9979.el9.x86 3.1 MB/s | 4.7 MB 00:01 2024-03-28T16:05:26.387 INFO:teuthology.orchestra.run.smithi083.stdout:(12/33): python3-cryptography-36.0.1-4.el9.x86_ 8.6 MB/s | 1.1 MB 00:00 2024-03-28T16:05:26.412 INFO:teuthology.orchestra.run.smithi083.stdout:(13/33): python3-ply-3.11-14.el9.noarch.rpm 774 kB/s | 103 kB 00:00 2024-03-28T16:05:26.437 INFO:teuthology.orchestra.run.smithi083.stdout:(14/33): python3-pycparser-2.20-6.el9.noarch.rp 1.5 MB/s | 124 kB 00:00 2024-03-28T16:05:26.463 INFO:teuthology.orchestra.run.smithi083.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-28T16:05:26.604 INFO:teuthology.orchestra.run.smithi083.stdout:(16/33): python3-requests-2.25.1-8.el9.noarch.r 650 kB/s | 125 kB 00:00 2024-03-28T16:05:26.663 INFO:teuthology.orchestra.run.smithi083.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 193 kB/s | 43 kB 00:00 2024-03-28T16:05:26.763 INFO:teuthology.orchestra.run.smithi083.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 572 kB/s | 172 kB 00:00 2024-03-28T16:05:26.806 INFO:teuthology.orchestra.run.smithi083.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-03-28T16:05:26.830 INFO:teuthology.orchestra.run.smithi083.stdout:(20/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 64 kB/s | 11 kB 00:00 2024-03-28T16:05:26.856 INFO:teuthology.orchestra.run.smithi083.stdout:(21/33): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-03-28T16:05:26.881 INFO:teuthology.orchestra.run.smithi083.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-03-28T16:05:26.898 INFO:teuthology.orchestra.run.smithi083.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 290 kB/s | 19 kB 00:00 2024-03-28T16:05:26.940 INFO:teuthology.orchestra.run.smithi083.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 235 kB/s | 20 kB 00:00 2024-03-28T16:05:27.032 INFO:teuthology.orchestra.run.smithi083.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 307 kB/s | 46 kB 00:00 2024-03-28T16:05:27.057 INFO:teuthology.orchestra.run.smithi083.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 495 kB/s | 79 kB 00:00 2024-03-28T16:05:27.124 INFO:teuthology.orchestra.run.smithi083.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-03-28T16:05:27.149 INFO:teuthology.orchestra.run.smithi083.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 140 kB/s | 16 kB 00:00 2024-03-28T16:05:27.183 INFO:teuthology.orchestra.run.smithi083.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 716 kB/s | 90 kB 00:00 2024-03-28T16:05:27.250 INFO:teuthology.orchestra.run.smithi083.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 284 kB/s | 36 kB 00:00 2024-03-28T16:05:27.275 INFO:teuthology.orchestra.run.smithi083.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-28T16:05:27.334 INFO:teuthology.orchestra.run.smithi083.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 239 kB/s | 20 kB 00:00 2024-03-28T16:05:27.359 INFO:teuthology.orchestra.run.smithi083.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2024-03-28T16:05:27.365 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:05:27.365 INFO:teuthology.orchestra.run.smithi083.stdout:Total 9.6 MB/s | 29 MB 00:03 2024-03-28T16:05:27.379 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:05:27.420 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:05:27.420 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:05:27.767 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:05:27.768 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:05:29.029 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:05:29.078 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-28T16:05:29.110 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-28T16:05:29.143 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-28T16:05:29.234 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-28T16:05:29.237 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1053.ge58b9979.el9.x86_64 5/6 2024-03-28T16:05:29.313 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-28T16:05:29.315 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-test-2:18.2.2-1053.ge58b9979.el9.x86_64 6/6 2024-03-28T16:05:29.363 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-28T16:05:29.443 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-28T16:05:29.497 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-28T16:05:29.552 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-28T16:05:29.600 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-28T16:05:29.656 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-28T16:05:29.840 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-28T16:05:29.898 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-28T16:05:29.935 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-28T16:05:29.971 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-28T16:05:30.001 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-28T16:05:30.078 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-28T16:05:30.120 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-28T16:05:30.154 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-28T16:05:30.194 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-28T16:05:30.229 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-28T16:05:30.263 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-28T16:05:30.351 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-28T16:05:30.400 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-28T16:05:30.475 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-28T16:05:30.610 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-28T16:05:30.727 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-28T16:05:30.881 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1053.ge58b9979.el9. 28/33 2024-03-28T16:05:30.908 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-2:18.2.2-1053.ge58b9979.el9.x86_64 29/33 2024-03-28T16:05:30.933 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1053.ge58b9979.el9.x86_64 29/33 2024-03-28T16:05:30.934 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T16:05:30.934 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T16:05:30.934 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T16:05:30.934 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T16:05:30.934 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:31.607 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-test-2:18.2.2-1053.ge58b9979.el9.x86_64 6/6 2024-03-28T16:05:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-test-2:18.2.2-1053.ge58b9979.el9.x86_64 1/6 2024-03-28T16:05:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephsqlite-2:18.2.2-1053.ge58b9979.el9.x86_64 2/6 2024-03-28T16:05:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-28T16:05:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-28T16:05:31.608 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-28T16:05:31.975 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-28T16:05:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:31.976 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:05:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: jq-1.6-16.el9.x86_64 2024-03-28T16:05:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T16:05:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T16:05:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T16:05:31.976 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:31.976 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:05:32.284 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph 2024-03-28T16:05:32.351 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-osd-2:18.2.2-1053.ge58b9979.el9.x86_64 30/33 2024-03-28T16:05:32.378 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-osd-2:18.2.2-1053.ge58b9979.el9.x86_64 30/33 2024-03-28T16:05:32.378 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T16:05:32.378 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T16:05:32.378 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T16:05:32.378 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T16:05:32.378 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:32.818 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mon-2:18.2.2-1053.ge58b9979.el9.x86_64 31/33 2024-03-28T16:05:32.841 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mon-2:18.2.2-1053.ge58b9979.el9.x86_64 31/33 2024-03-28T16:05:32.841 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T16:05:32.841 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T16:05:32.841 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T16:05:32.842 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T16:05:32.842 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:32.849 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:41 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:05:33.007 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:05:33.009 INFO:teuthology.orchestra.run.smithi007.stdout:========================================================================================= 2024-03-28T16:05:33.009 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-28T16:05:33.009 INFO:teuthology.orchestra.run.smithi007.stdout:========================================================================================= 2024-03-28T16:05:33.009 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:05:33.009 INFO:teuthology.orchestra.run.smithi007.stdout: ceph x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 6.5 k 2024-03-28T16:05:33.010 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-28T16:05:33.010 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 2.1 M 2024-03-28T16:05:33.010 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 1.5 M 2024-03-28T16:05:33.010 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1053.ge58b9979.el9 ceph-noarch 245 k 2024-03-28T16:05:33.010 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 4.7 M 2024-03-28T16:05:33.010 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 17 M 2024-03-28T16:05:33.010 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T16:05:33.010 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T16:05:33.010 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T16:05:33.010 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T16:05:33.010 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T16:05:33.010 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T16:05:33.010 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T16:05:33.010 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T16:05:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T16:05:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T16:05:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T16:05:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T16:05:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T16:05:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T16:05:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T16:05:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T16:05:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T16:05:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T16:05:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T16:05:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T16:05:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T16:05:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T16:05:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T16:05:33.011 INFO:teuthology.orchestra.run.smithi007.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T16:05:33.012 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T16:05:33.012 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T16:05:33.012 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T16:05:33.012 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:33.012 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:05:33.012 INFO:teuthology.orchestra.run.smithi007.stdout:========================================================================================= 2024-03-28T16:05:33.012 INFO:teuthology.orchestra.run.smithi007.stdout:Install 33 Packages 2024-03-28T16:05:33.012 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:33.012 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 29 M 2024-03-28T16:05:33.013 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 104 M 2024-03-28T16:05:33.013 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:05:33.044 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mds-2:18.2.2-1053.ge58b9979.el9.x86_64 32/33 2024-03-28T16:05:33.067 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mds-2:18.2.2-1053.ge58b9979.el9.x86_64 32/33 2024-03-28T16:05:33.067 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T16:05:33.067 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T16:05:33.067 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T16:05:33.068 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T16:05:33.068 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:33.132 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-2:18.2.2-1053.ge58b9979.el9.x86_64 33/33 2024-03-28T16:05:33.622 INFO:teuthology.orchestra.run.smithi007.stdout:(1/33): ceph-18.2.2-1053.ge58b9979.el9.x86_64.r 39 kB/s | 6.5 kB 00:00 2024-03-28T16:05:34.064 INFO:teuthology.orchestra.run.smithi007.stdout:(2/33): ceph-mon-18.2.2-1053.ge58b9979.el9.x86_ 11 MB/s | 4.7 MB 00:00 2024-03-28T16:05:34.114 INFO:teuthology.orchestra.run.smithi007.stdout:(3/33): ceph-mds-18.2.2-1053.ge58b9979.el9.x86_ 3.2 MB/s | 2.1 MB 00:00 2024-03-28T16:05:34.190 INFO:teuthology.orchestra.run.smithi007.stdout:(4/33): ceph-mgr-18.2.2-1053.ge58b9979.el9.x86_ 2.0 MB/s | 1.5 MB 00:00 2024-03-28T16:05:34.223 INFO:teuthology.orchestra.run.smithi007.stdout:(5/33): ceph-mgr-modules-core-18.2.2-1053.ge58b 2.2 MB/s | 245 kB 00:00 2024-03-28T16:05:34.332 INFO:teuthology.orchestra.run.smithi007.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 377 kB/s | 32 kB 00:00 2024-03-28T16:05:34.357 INFO:teuthology.orchestra.run.smithi007.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 861 kB/s | 144 kB 00:00 2024-03-28T16:05:34.374 INFO:teuthology.orchestra.run.smithi007.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-28T16:05:34.391 INFO:teuthology.orchestra.run.smithi007.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-28T16:05:34.425 INFO:teuthology.orchestra.run.smithi007.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-28T16:05:34.483 INFO:teuthology.orchestra.run.smithi007.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 12 MB/s | 1.1 MB 00:00 2024-03-28T16:05:34.509 INFO:teuthology.orchestra.run.smithi007.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-28T16:05:34.534 INFO:teuthology.orchestra.run.smithi007.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-28T16:05:34.559 INFO:teuthology.orchestra.run.smithi007.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-28T16:05:34.701 INFO:teuthology.orchestra.run.smithi007.stdout:(15/33): python3-requests-2.25.1-8.el9.noarch.r 748 kB/s | 125 kB 00:00 2024-03-28T16:05:34.902 INFO:teuthology.orchestra.run.smithi007.stdout:(16/33): ceph-osd-18.2.2-1053.ge58b9979.el9.x86 20 MB/s | 17 MB 00:00 2024-03-28T16:05:34.916 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-2:18.2.2-1053.ge58b9979.el9.x86_64 33/33 2024-03-28T16:05:34.916 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-2:18.2.2-1053.ge58b9979.el9.x86_64 1/33 2024-03-28T16:05:34.916 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mds-2:18.2.2-1053.ge58b9979.el9.x86_64 2/33 2024-03-28T16:05:34.916 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-2:18.2.2-1053.ge58b9979.el9.x86_64 3/33 2024-03-28T16:05:34.917 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mon-2:18.2.2-1053.ge58b9979.el9.x86_64 4/33 2024-03-28T16:05:34.917 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-osd-2:18.2.2-1053.ge58b9979.el9.x86_64 5/33 2024-03-28T16:05:34.917 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1053.ge58b9979.el9. 6/33 2024-03-28T16:05:34.917 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-28T16:05:34.917 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-28T16:05:34.917 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-28T16:05:34.917 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-28T16:05:34.917 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-28T16:05:34.917 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-28T16:05:34.917 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-28T16:05:34.917 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-28T16:05:34.917 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-28T16:05:34.917 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-28T16:05:34.917 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-28T16:05:34.918 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-28T16:05:34.918 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-28T16:05:34.918 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-28T16:05:34.918 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-28T16:05:34.918 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-28T16:05:34.918 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-28T16:05:34.918 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-28T16:05:34.918 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-28T16:05:34.919 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-28T16:05:34.919 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-28T16:05:34.919 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-28T16:05:34.919 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-28T16:05:34.919 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-28T16:05:34.919 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-28T16:05:34.919 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-28T16:05:34.952 INFO:teuthology.orchestra.run.smithi007.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 111 kB/s | 43 kB 00:00 2024-03-28T16:05:35.027 INFO:teuthology.orchestra.run.smithi007.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2024-03-28T16:05:35.111 INFO:teuthology.orchestra.run.smithi007.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 212 kB/s | 18 kB 00:00 2024-03-28T16:05:35.194 INFO:teuthology.orchestra.run.smithi007.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 278 kB/s | 23 kB 00:00 2024-03-28T16:05:35.245 INFO:teuthology.orchestra.run.smithi007.stdout:(21/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 317 kB/s | 172 kB 00:00 2024-03-28T16:05:35.303 INFO:teuthology.orchestra.run.smithi007.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2024-03-28T16:05:35.329 INFO:teuthology.orchestra.run.smithi007.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 235 kB/s | 20 kB 00:00 2024-03-28T16:05:35.362 INFO:teuthology.orchestra.run.smithi007.stdout:(24/33): python3-cherrypy-18.6.1-2.el9.noarch.r 826 kB/s | 358 kB 00:00 2024-03-28T16:05:35.404 INFO:teuthology.orchestra.run.smithi007.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 460 kB/s | 46 kB 00:00 2024-03-28T16:05:35.429 INFO:teuthology.orchestra.run.smithi007.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 785 kB/s | 79 kB 00:00 2024-03-28T16:05:35.463 INFO:teuthology.orchestra.run.smithi007.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-03-28T16:05:35.488 INFO:teuthology.orchestra.run.smithi007.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 195 kB/s | 16 kB 00:00 2024-03-28T16:05:35.530 INFO:teuthology.orchestra.run.smithi007.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 895 kB/s | 90 kB 00:00 2024-03-28T16:05:35.547 INFO:teuthology.orchestra.run.smithi007.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 427 kB/s | 36 kB 00:00 2024-03-28T16:05:35.614 INFO:teuthology.orchestra.run.smithi007.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 300 kB/s | 20 kB 00:00 2024-03-28T16:05:35.664 INFO:teuthology.orchestra.run.smithi007.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-03-28T16:05:35.698 INFO:teuthology.orchestra.run.smithi007.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.5 MB/s | 427 kB 00:00 2024-03-28T16:05:35.704 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:05:35.704 INFO:teuthology.orchestra.run.smithi007.stdout:Total 11 MB/s | 29 MB 00:02 2024-03-28T16:05:35.718 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:05:35.763 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:05:35.763 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:05:35.895 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-28T16:05:35.895 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:35.895 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:05:35.895 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:35.895 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:35.895 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:35.895 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core-2:18.2.2-1053.ge58b9979.el9.noarch 2024-03-28T16:05:35.895 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:35.895 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:35.895 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T16:05:35.895 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T16:05:35.895 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T16:05:35.895 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T16:05:35.896 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T16:05:35.896 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T16:05:35.896 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T16:05:35.896 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T16:05:35.896 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T16:05:35.896 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T16:05:35.896 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T16:05:35.896 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T16:05:35.896 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T16:05:35.896 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T16:05:35.896 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T16:05:35.896 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T16:05:35.896 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T16:05:35.897 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T16:05:35.897 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T16:05:35.897 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T16:05:35.897 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T16:05:35.897 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T16:05:35.897 INFO:teuthology.orchestra.run.smithi083.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T16:05:35.897 INFO:teuthology.orchestra.run.smithi083.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T16:05:35.897 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T16:05:35.897 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T16:05:35.897 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T16:05:35.897 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:35.897 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:05:36.145 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:05:36.145 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:05:36.251 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-base 2024-03-28T16:05:36.822 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:56 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:05:36.917 INFO:teuthology.orchestra.run.smithi083.stdout:Package ceph-base-2:18.2.2-1053.ge58b9979.el9.x86_64 is already installed. 2024-03-28T16:05:36.968 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:05:36.969 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-28T16:05:36.969 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:05:37.030 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install cephadm 2024-03-28T16:05:37.541 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:05:37.580 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:57 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:05:37.584 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-28T16:05:37.617 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-28T16:05:37.650 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-28T16:05:37.727 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:05:37.727 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:05:37.728 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-28T16:05:37.728 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:05:37.728 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:05:37.728 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm noarch 2:18.2.2-1053.ge58b9979.el9 ceph-noarch 224 k 2024-03-28T16:05:37.728 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:37.728 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:05:37.728 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:05:37.728 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:05:37.728 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:37.728 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 224 k 2024-03-28T16:05:37.728 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 216 k 2024-03-28T16:05:37.728 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:05:37.745 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-28T16:05:37.823 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-28T16:05:37.870 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-28T16:05:37.951 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-28T16:05:37.985 INFO:teuthology.orchestra.run.smithi083.stdout:cephadm-18.2.2-1053.ge58b9979.el9.noarch.rpm 874 kB/s | 224 kB 00:00 2024-03-28T16:05:37.985 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:05:37.986 INFO:teuthology.orchestra.run.smithi083.stdout:Total 869 kB/s | 224 kB 00:00 2024-03-28T16:05:37.986 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:05:37.991 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:05:37.991 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:05:37.993 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:05:37.994 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:05:38.005 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-28T16:05:38.042 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-28T16:05:38.066 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:05:38.101 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-28T16:05:38.164 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-28T16:05:38.365 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-28T16:05:38.430 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-28T16:05:38.475 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-28T16:05:38.490 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephadm-2:18.2.2-1053.ge58b9979.el9.noarch 1/1 2024-03-28T16:05:38.520 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-28T16:05:38.549 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-28T16:05:38.567 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cephadm-2:18.2.2-1053.ge58b9979.el9.noarch 1/1 2024-03-28T16:05:38.629 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-28T16:05:38.668 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-28T16:05:38.702 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-28T16:05:38.743 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-28T16:05:38.786 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-28T16:05:38.821 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-28T16:05:38.913 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-28T16:05:38.938 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephadm-2:18.2.2-1053.ge58b9979.el9.noarch 1/1 2024-03-28T16:05:38.956 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-28T16:05:39.035 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-28T16:05:39.174 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-28T16:05:39.206 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cephadm-2:18.2.2-1053.ge58b9979.el9.noarch 1/1 2024-03-28T16:05:39.207 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:39.207 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:05:39.207 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm-2:18.2.2-1053.ge58b9979.el9.noarch 2024-03-28T16:05:39.207 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:39.207 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:05:39.298 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-28T16:05:39.424 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-immutable-object-cache 2024-03-28T16:05:39.461 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1053.ge58b9979.el9. 28/33 2024-03-28T16:05:39.489 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-2:18.2.2-1053.ge58b9979.el9.x86_64 29/33 2024-03-28T16:05:39.514 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1053.ge58b9979.el9.x86_64 29/33 2024-03-28T16:05:39.514 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T16:05:39.514 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T16:05:39.514 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T16:05:39.514 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T16:05:39.514 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:39.990 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:59 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:05:40.144 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:05:40.144 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:05:40.145 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:05:40.145 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:05:40.145 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:05:40.145 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 143 k 2024-03-28T16:05:40.145 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:40.145 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:05:40.145 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:05:40.145 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:05:40.145 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:40.145 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 143 k 2024-03-28T16:05:40.145 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 395 k 2024-03-28T16:05:40.145 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:05:40.536 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-immutable-object-cache-18.2.2-1053.ge58b99 365 kB/s | 143 kB 00:00 2024-03-28T16:05:40.537 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:05:40.537 INFO:teuthology.orchestra.run.smithi083.stdout:Total 363 kB/s | 143 kB 00:00 2024-03-28T16:05:40.537 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:05:40.545 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:05:40.545 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:05:40.598 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:05:40.598 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:05:40.732 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:05:40.804 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1053.ge58b9979. 1/1 2024-03-28T16:05:40.828 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1053.ge58b9979. 1/1 2024-03-28T16:05:40.828 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T16:05:40.828 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T16:05:40.828 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:41.025 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-osd-2:18.2.2-1053.ge58b9979.el9.x86_64 30/33 2024-03-28T16:05:41.053 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-osd-2:18.2.2-1053.ge58b9979.el9.x86_64 30/33 2024-03-28T16:05:41.053 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T16:05:41.053 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T16:05:41.053 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T16:05:41.054 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T16:05:41.054 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:41.518 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mon-2:18.2.2-1053.ge58b9979.el9.x86_64 31/33 2024-03-28T16:05:41.541 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mon-2:18.2.2-1053.ge58b9979.el9.x86_64 31/33 2024-03-28T16:05:41.541 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T16:05:41.541 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T16:05:41.541 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T16:05:41.541 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T16:05:41.541 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:41.760 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mds-2:18.2.2-1053.ge58b9979.el9.x86_64 32/33 2024-03-28T16:05:41.783 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mds-2:18.2.2-1053.ge58b9979.el9.x86_64 32/33 2024-03-28T16:05:41.783 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T16:05:41.783 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T16:05:41.783 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T16:05:41.783 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T16:05:41.783 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:41.836 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1053.ge58b9979. 1/1 2024-03-28T16:05:41.836 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:41.836 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:05:41.836 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:41.836 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:41.837 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:05:41.852 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-2:18.2.2-1053.ge58b9979.el9.x86_64 33/33 2024-03-28T16:05:42.117 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr 2024-03-28T16:05:42.687 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:02 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:05:42.784 INFO:teuthology.orchestra.run.smithi083.stdout:Package ceph-mgr-2:18.2.2-1053.ge58b9979.el9.x86_64 is already installed. 2024-03-28T16:05:42.835 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:05:42.836 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-28T16:05:42.836 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:05:42.897 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-dashboard 2024-03-28T16:05:43.452 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:03 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:05:43.602 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:05:43.602 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:05:43.603 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-28T16:05:43.603 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:05:43.603 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:05:43.603 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1053.ge58b9979.el9 ceph-noarch 1.8 M 2024-03-28T16:05:43.603 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-28T16:05:43.603 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1053.ge58b9979.el9 ceph-noarch 23 k 2024-03-28T16:05:43.603 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1053.ge58b9979.el9 ceph-noarch 15 k 2024-03-28T16:05:43.603 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T16:05:43.603 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T16:05:43.603 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:43.603 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:05:43.603 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:05:43.603 INFO:teuthology.orchestra.run.smithi083.stdout:Install 5 Packages 2024-03-28T16:05:43.603 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:43.604 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 2.0 M 2024-03-28T16:05:43.604 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 8.6 M 2024-03-28T16:05:43.604 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:05:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-2:18.2.2-1053.ge58b9979.el9.x86_64 33/33 2024-03-28T16:05:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-2:18.2.2-1053.ge58b9979.el9.x86_64 1/33 2024-03-28T16:05:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mds-2:18.2.2-1053.ge58b9979.el9.x86_64 2/33 2024-03-28T16:05:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-2:18.2.2-1053.ge58b9979.el9.x86_64 3/33 2024-03-28T16:05:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mon-2:18.2.2-1053.ge58b9979.el9.x86_64 4/33 2024-03-28T16:05:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-osd-2:18.2.2-1053.ge58b9979.el9.x86_64 5/33 2024-03-28T16:05:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1053.ge58b9979.el9. 6/33 2024-03-28T16:05:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-28T16:05:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-28T16:05:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-28T16:05:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-28T16:05:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-28T16:05:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-28T16:05:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-28T16:05:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-28T16:05:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-28T16:05:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-28T16:05:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-28T16:05:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-28T16:05:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-28T16:05:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-28T16:05:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-28T16:05:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-28T16:05:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-28T16:05:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-28T16:05:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-28T16:05:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-28T16:05:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-28T16:05:43.641 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-28T16:05:43.641 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-28T16:05:43.641 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-28T16:05:43.641 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-28T16:05:43.641 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-28T16:05:44.074 INFO:teuthology.orchestra.run.smithi083.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1053.ge58 91 kB/s | 23 kB 00:00 2024-03-28T16:05:44.099 INFO:teuthology.orchestra.run.smithi083.stdout:(2/5): ceph-prometheus-alerts-18.2.2-1053.ge58b 53 kB/s | 15 kB 00:00 2024-03-28T16:05:44.350 INFO:teuthology.orchestra.run.smithi083.stdout:(3/5): ceph-mgr-dashboard-18.2.2-1053.ge58b9979 3.4 MB/s | 1.8 MB 00:00 2024-03-28T16:05:44.393 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-28T16:05:44.393 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:44.393 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:05:44.393 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:44.394 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:44.394 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:44.394 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core-2:18.2.2-1053.ge58b9979.el9.noarch 2024-03-28T16:05:44.394 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:44.394 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:44.395 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T16:05:44.395 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T16:05:44.395 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T16:05:44.395 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T16:05:44.395 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T16:05:44.395 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T16:05:44.395 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T16:05:44.395 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T16:05:44.395 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T16:05:44.395 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T16:05:44.395 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T16:05:44.395 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T16:05:44.396 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T16:05:44.396 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T16:05:44.396 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T16:05:44.396 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T16:05:44.396 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T16:05:44.396 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T16:05:44.396 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T16:05:44.396 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T16:05:44.396 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T16:05:44.396 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T16:05:44.397 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T16:05:44.397 INFO:teuthology.orchestra.run.smithi007.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T16:05:44.397 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T16:05:44.397 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T16:05:44.397 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T16:05:44.398 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:44.398 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:05:44.575 INFO:teuthology.orchestra.run.smithi083.stdout:(4/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 61 kB/s | 31 kB 00:00 2024-03-28T16:05:44.789 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-base 2024-03-28T16:05:44.800 INFO:teuthology.orchestra.run.smithi083.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 269 kB/s | 188 kB 00:00 2024-03-28T16:05:44.803 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:05:44.803 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-03-28T16:05:44.803 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:05:44.810 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:05:44.810 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:05:44.865 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:05:44.865 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:05:45.103 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:05:45.197 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-03-28T16:05:45.230 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-03-28T16:05:45.258 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1053.ge58b9979.el9.n 3/5 2024-03-28T16:05:45.359 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:54 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:05:45.459 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-base-2:18.2.2-1053.ge58b9979.el9.x86_64 is already installed. 2024-03-28T16:05:45.513 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:05:45.514 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-28T16:05:45.514 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:05:45.514 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1053.ge58b9979.el9. 4/5 2024-03-28T16:05:45.553 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1053.ge58b9979.el9.noarc 5/5 2024-03-28T16:05:45.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install cephadm 2024-03-28T16:05:46.036 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1053.ge58b9979.el9.noarc 5/5 2024-03-28T16:05:46.036 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1053.ge58b9979.el9. 1/5 2024-03-28T16:05:46.036 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1053.ge58b9979.el9.noarc 2/5 2024-03-28T16:05:46.036 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1053.ge58b9979.el9.n 3/5 2024-03-28T16:05:46.036 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-03-28T16:05:46.162 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:55 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:05:46.315 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:05:46.316 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:05:46.316 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-28T16:05:46.316 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:05:46.316 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:05:46.316 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm noarch 2:18.2.2-1053.ge58b9979.el9 ceph-noarch 224 k 2024-03-28T16:05:46.316 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:46.316 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:05:46.316 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:05:46.316 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:05:46.317 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:46.317 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 224 k 2024-03-28T16:05:46.317 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 216 k 2024-03-28T16:05:46.317 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:05:46.317 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-03-28T16:05:46.317 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:46.317 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:05:46.317 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards-2:18.2.2-1053.ge58b9979.el9.noarch 2024-03-28T16:05:46.317 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard-2:18.2.2-1053.ge58b9979.el9.noarch 2024-03-28T16:05:46.318 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts-2:18.2.2-1053.ge58b9979.el9.noarch 2024-03-28T16:05:46.318 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T16:05:46.318 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T16:05:46.318 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:46.318 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:05:46.508 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-28T16:05:46.572 INFO:teuthology.orchestra.run.smithi007.stdout:cephadm-18.2.2-1053.ge58b9979.el9.noarch.rpm 880 kB/s | 224 kB 00:00 2024-03-28T16:05:46.572 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:05:46.572 INFO:teuthology.orchestra.run.smithi007.stdout:Total 875 kB/s | 224 kB 00:00 2024-03-28T16:05:46.573 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:05:46.578 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:05:46.578 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:05:46.581 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:05:46.581 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:05:46.657 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:05:47.062 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:07 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:05:47.103 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:18.2.2-1053.ge58b9979.el9.noarch 1/1 2024-03-28T16:05:47.197 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephadm-2:18.2.2-1053.ge58b9979.el9.noarch 1/1 2024-03-28T16:05:47.211 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:05:47.213 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================================ 2024-03-28T16:05:47.213 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-28T16:05:47.213 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================================ 2024-03-28T16:05:47.213 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:05:47.213 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1053.ge58b9979.el9 ceph-noarch 7.4 M 2024-03-28T16:05:47.213 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-28T16:05:47.213 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T16:05:47.213 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T16:05:47.213 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T16:05:47.213 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T16:05:47.214 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T16:05:47.214 INFO:teuthology.orchestra.run.smithi083.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T16:05:47.214 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T16:05:47.214 INFO:teuthology.orchestra.run.smithi083.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T16:05:47.214 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T16:05:47.214 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T16:05:47.214 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T16:05:47.214 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:47.214 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:05:47.214 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================================ 2024-03-28T16:05:47.214 INFO:teuthology.orchestra.run.smithi083.stdout:Install 12 Packages 2024-03-28T16:05:47.214 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:47.215 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 37 M 2024-03-28T16:05:47.215 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 209 M 2024-03-28T16:05:47.215 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:05:47.546 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:18.2.2-1053.ge58b9979.el9.noarch 1/1 2024-03-28T16:05:47.756 INFO:teuthology.orchestra.run.smithi083.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 194 kB/s | 32 kB 00:00 2024-03-28T16:05:47.784 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephadm-2:18.2.2-1053.ge58b9979.el9.noarch 1/1 2024-03-28T16:05:47.784 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:47.784 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:05:47.784 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm-2:18.2.2-1053.ge58b9979.el9.noarch 2024-03-28T16:05:47.784 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:47.784 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:05:47.789 INFO:teuthology.orchestra.run.smithi083.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 534 kB/s | 17 kB 00:00 2024-03-28T16:05:47.823 INFO:teuthology.orchestra.run.smithi083.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-28T16:05:47.881 INFO:teuthology.orchestra.run.smithi083.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-03-28T16:05:48.082 INFO:teuthology.orchestra.run.smithi083.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-03-28T16:05:48.093 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-immutable-object-cache 2024-03-28T16:05:48.149 INFO:teuthology.orchestra.run.smithi083.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-28T16:05:48.174 INFO:teuthology.orchestra.run.smithi083.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-28T16:05:48.358 INFO:teuthology.orchestra.run.smithi083.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.3 MB/s | 803 kB 00:00 2024-03-28T16:05:48.516 INFO:teuthology.orchestra.run.smithi083.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-28T16:05:48.550 INFO:teuthology.orchestra.run.smithi083.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-28T16:05:48.642 INFO:teuthology.orchestra.run.smithi083.stdout:(11/12): ceph-mgr-diskprediction-local-18.2.2-1 7.1 MB/s | 7.4 MB 00:01 2024-03-28T16:05:48.665 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:57 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:05:48.725 INFO:teuthology.orchestra.run.smithi083.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-28T16:05:48.727 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:05:48.727 INFO:teuthology.orchestra.run.smithi083.stdout:Total 25 MB/s | 37 MB 00:01 2024-03-28T16:05:48.766 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:05:48.789 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:05:48.789 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:05:48.821 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:05:48.822 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:05:48.822 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:05:48.822 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:05:48.822 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:05:48.822 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 143 k 2024-03-28T16:05:48.823 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:48.823 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:05:48.823 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:05:48.823 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:05:48.823 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:48.823 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 143 k 2024-03-28T16:05:48.823 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 395 k 2024-03-28T16:05:48.823 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:05:49.136 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-immutable-object-cache-18.2.2-1053.ge58b99 456 kB/s | 143 kB 00:00 2024-03-28T16:05:49.137 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:05:49.137 INFO:teuthology.orchestra.run.smithi007.stdout:Total 453 kB/s | 143 kB 00:00 2024-03-28T16:05:49.137 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:05:49.145 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:05:49.145 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:05:49.201 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:05:49.201 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:05:49.227 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:05:49.227 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:05:49.468 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:05:49.537 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1053.ge58b9979. 1/1 2024-03-28T16:05:49.561 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1053.ge58b9979. 1/1 2024-03-28T16:05:49.561 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T16:05:49.561 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T16:05:49.561 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:49.928 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:05:50.006 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-28T16:05:50.038 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-28T16:05:50.139 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-28T16:05:50.183 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-28T16:05:50.361 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-28T16:05:50.399 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-28T16:05:50.487 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-28T16:05:50.674 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1053.ge58b9979. 1/1 2024-03-28T16:05:50.675 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:50.675 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:05:50.675 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:05:50.675 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:50.675 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:05:50.940 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr 2024-03-28T16:05:51.163 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-28T16:05:51.277 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-28T16:05:51.517 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:00 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:05:51.617 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-mgr-2:18.2.2-1053.ge58b9979.el9.x86_64 is already installed. 2024-03-28T16:05:51.670 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:05:51.671 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-28T16:05:51.672 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:05:51.747 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-dashboard 2024-03-28T16:05:52.315 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:01 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:05:52.471 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:05:52.472 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:05:52.472 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-28T16:05:52.472 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:05:52.472 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:05:52.472 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1053.ge58b9979.el9 ceph-noarch 1.8 M 2024-03-28T16:05:52.472 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-28T16:05:52.472 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1053.ge58b9979.el9 ceph-noarch 23 k 2024-03-28T16:05:52.472 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1053.ge58b9979.el9 ceph-noarch 15 k 2024-03-28T16:05:52.472 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T16:05:52.472 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T16:05:52.473 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:52.473 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:05:52.473 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:05:52.473 INFO:teuthology.orchestra.run.smithi007.stdout:Install 5 Packages 2024-03-28T16:05:52.473 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:52.473 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 2.0 M 2024-03-28T16:05:52.474 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 8.6 M 2024-03-28T16:05:52.474 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:05:52.522 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-28T16:05:53.016 INFO:teuthology.orchestra.run.smithi007.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1053.ge58 128 kB/s | 23 kB 00:00 2024-03-28T16:05:53.032 INFO:teuthology.orchestra.run.smithi007.stdout:(2/5): ceph-prometheus-alerts-18.2.2-1053.ge58b 74 kB/s | 15 kB 00:00 2024-03-28T16:05:53.082 INFO:teuthology.orchestra.run.smithi007.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 459 kB/s | 31 kB 00:00 2024-03-28T16:05:53.133 INFO:teuthology.orchestra.run.smithi007.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-03-28T16:05:53.317 INFO:teuthology.orchestra.run.smithi007.stdout:(5/5): ceph-mgr-dashboard-18.2.2-1053.ge58b9979 3.7 MB/s | 1.8 MB 00:00 2024-03-28T16:05:53.318 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:05:53.319 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-03-28T16:05:53.319 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:05:53.326 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:05:53.326 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:05:53.389 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:05:53.389 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:05:53.545 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-28T16:05:53.574 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1053.ge58b9 12/12 2024-03-28T16:05:53.626 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:05:53.715 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-03-28T16:05:53.755 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-03-28T16:05:53.784 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1053.ge58b9979.el9.n 3/5 2024-03-28T16:05:54.034 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1053.ge58b9979.el9. 4/5 2024-03-28T16:05:54.069 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1053.ge58b9979.el9.noarc 5/5 2024-03-28T16:05:54.555 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1053.ge58b9979.el9.noarc 5/5 2024-03-28T16:05:54.555 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1053.ge58b9979.el9. 1/5 2024-03-28T16:05:54.555 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1053.ge58b9979.el9.noarc 2/5 2024-03-28T16:05:54.555 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1053.ge58b9979.el9.n 3/5 2024-03-28T16:05:54.555 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-03-28T16:05:54.842 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-03-28T16:05:54.842 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:54.842 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:05:54.842 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards-2:18.2.2-1053.ge58b9979.el9.noarch 2024-03-28T16:05:54.842 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard-2:18.2.2-1053.ge58b9979.el9.noarch 2024-03-28T16:05:54.842 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts-2:18.2.2-1053.ge58b9979.el9.noarch 2024-03-28T16:05:54.842 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T16:05:54.843 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T16:05:54.843 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:54.843 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:05:55.063 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-28T16:05:55.630 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:04 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:05:55.784 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:05:55.785 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2024-03-28T16:05:55.785 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-28T16:05:55.786 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2024-03-28T16:05:55.786 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:05:55.786 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1053.ge58b9979.el9 ceph-noarch 7.4 M 2024-03-28T16:05:55.786 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-28T16:05:55.786 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T16:05:55.786 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T16:05:55.786 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T16:05:55.786 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T16:05:55.786 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T16:05:55.786 INFO:teuthology.orchestra.run.smithi007.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T16:05:55.786 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T16:05:55.786 INFO:teuthology.orchestra.run.smithi007.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T16:05:55.787 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T16:05:55.787 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T16:05:55.787 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T16:05:55.787 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:55.787 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:05:55.787 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2024-03-28T16:05:55.787 INFO:teuthology.orchestra.run.smithi007.stdout:Install 12 Packages 2024-03-28T16:05:55.787 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:05:55.788 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 37 M 2024-03-28T16:05:55.788 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 209 M 2024-03-28T16:05:55.788 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:05:56.117 INFO:teuthology.orchestra.run.smithi007.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 230 kB/s | 32 kB 00:00 2024-03-28T16:05:56.148 INFO:teuthology.orchestra.run.smithi007.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 530 kB/s | 17 kB 00:00 2024-03-28T16:05:56.215 INFO:teuthology.orchestra.run.smithi007.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 526 kB/s | 34 kB 00:00 2024-03-28T16:05:56.216 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1053.ge58b9 12/12 2024-03-28T16:05:56.216 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1053.ge58b9 1/12 2024-03-28T16:05:56.216 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-28T16:05:56.216 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-28T16:05:56.216 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-28T16:05:56.216 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-28T16:05:56.217 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-28T16:05:56.217 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-28T16:05:56.217 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-28T16:05:56.217 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-28T16:05:56.217 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-28T16:05:56.217 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-28T16:05:56.274 INFO:teuthology.orchestra.run.smithi007.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-03-28T16:05:56.685 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-28T16:05:56.686 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:56.686 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:05:56.686 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1053.ge58b9979.el9.noarch 2024-03-28T16:05:56.686 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T16:05:56.686 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T16:05:56.686 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T16:05:56.686 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T16:05:56.686 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T16:05:56.686 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T16:05:56.686 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T16:05:56.686 INFO:teuthology.orchestra.run.smithi083.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T16:05:56.686 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T16:05:56.687 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T16:05:56.687 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T16:05:56.687 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:56.687 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:05:57.015 INFO:teuthology.orchestra.run.smithi007.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 5.9 MB/s | 4.7 MB 00:00 2024-03-28T16:05:57.070 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-rook 2024-03-28T16:05:57.149 INFO:teuthology.orchestra.run.smithi007.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.5 MB/s | 4.8 MB 00:00 2024-03-28T16:05:57.174 INFO:teuthology.orchestra.run.smithi007.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.3 MB/s | 217 kB 00:00 2024-03-28T16:05:57.631 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:17 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:05:57.781 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:05:57.782 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-28T16:05:57.782 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-28T16:05:57.782 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-28T16:05:57.782 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:05:57.782 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook noarch 2:18.2.2-1053.ge58b9979.el9 ceph-noarch 49 k 2024-03-28T16:05:57.782 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-28T16:05:57.782 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T16:05:57.782 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T16:05:57.782 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T16:05:57.783 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T16:05:57.783 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T16:05:57.783 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T16:05:57.783 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T16:05:57.783 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T16:05:57.783 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T16:05:57.783 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T16:05:57.783 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T16:05:57.783 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T16:05:57.783 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:57.783 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:05:57.783 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-28T16:05:57.783 INFO:teuthology.orchestra.run.smithi083.stdout:Install 13 Packages 2024-03-28T16:05:57.783 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:05:57.784 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 2.0 M 2024-03-28T16:05:57.784 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 26 M 2024-03-28T16:05:57.784 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:05:57.800 INFO:teuthology.orchestra.run.smithi007.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 803 kB 00:00 2024-03-28T16:05:58.033 INFO:teuthology.orchestra.run.smithi007.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 18 MB/s | 16 MB 00:00 2024-03-28T16:05:58.059 INFO:teuthology.orchestra.run.smithi007.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 759 kB/s | 195 kB 00:00 2024-03-28T16:05:58.075 INFO:teuthology.orchestra.run.smithi083.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 155 kB/s | 24 kB 00:00 2024-03-28T16:05:58.101 INFO:teuthology.orchestra.run.smithi083.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 99 kB/s | 18 kB 00:00 2024-03-28T16:05:58.218 INFO:teuthology.orchestra.run.smithi083.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.2 MB/s | 174 kB 00:00 2024-03-28T16:05:58.268 INFO:teuthology.orchestra.run.smithi083.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 792 kB/s | 132 kB 00:00 2024-03-28T16:05:58.318 INFO:teuthology.orchestra.run.smithi083.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.0 MB/s | 211 kB 00:00 2024-03-28T16:05:58.325 INFO:teuthology.orchestra.run.smithi007.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 841 kB/s | 245 kB 00:00 2024-03-28T16:05:58.343 INFO:teuthology.orchestra.run.smithi083.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 567 kB/s | 43 kB 00:00 2024-03-28T16:05:58.369 INFO:teuthology.orchestra.run.smithi083.stdout:(7/13): ceph-mgr-rook-18.2.2-1053.ge58b9979.el9 109 kB/s | 49 kB 00:00 2024-03-28T16:05:58.476 INFO:teuthology.orchestra.run.smithi007.stdout:(12/12): ceph-mgr-diskprediction-local-18.2.2-1 3.0 MB/s | 7.4 MB 00:02 2024-03-28T16:05:58.479 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:05:58.479 INFO:teuthology.orchestra.run.smithi007.stdout:Total 14 MB/s | 37 MB 00:02 2024-03-28T16:05:58.502 INFO:teuthology.orchestra.run.smithi083.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 175 kB/s | 32 kB 00:00 2024-03-28T16:05:58.523 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:05:58.544 INFO:teuthology.orchestra.run.smithi083.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 75 kB/s | 15 kB 00:00 2024-03-28T16:05:58.548 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:05:58.548 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:05:58.694 INFO:teuthology.orchestra.run.smithi083.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 394 kB/s | 59 kB 00:00 2024-03-28T16:05:58.720 INFO:teuthology.orchestra.run.smithi083.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 664 kB/s | 203 kB 00:00 2024-03-28T16:05:58.745 INFO:teuthology.orchestra.run.smithi083.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 4.2 MB/s | 1.0 MB 00:00 2024-03-28T16:05:58.787 INFO:teuthology.orchestra.run.smithi083.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 982 kB/s | 90 kB 00:00 2024-03-28T16:05:58.791 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:05:58.791 INFO:teuthology.orchestra.run.smithi083.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-03-28T16:05:58.791 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:05:58.804 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:05:58.804 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:05:58.893 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:05:58.893 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:05:58.992 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:05:58.993 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:05:59.423 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:05:59.588 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-28T16:05:59.645 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-28T16:05:59.700 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-28T16:05:59.718 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:05:59.730 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-28T16:05:59.771 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-28T16:05:59.782 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-28T16:05:59.813 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-28T16:05:59.864 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-28T16:05:59.910 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-28T16:05:59.950 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-28T16:05:59.981 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-28T16:06:00.039 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-28T16:06:00.134 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-28T16:06:00.165 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-28T16:06:00.250 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-28T16:06:00.750 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-28T16:06:00.804 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-28T16:06:00.838 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-28T16:06:00.885 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-28T16:06:00.908 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-rook-2:18.2.2-1053.ge58b9979.el9.noarch 13/13 2024-03-28T16:06:00.973 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-28T16:06:01.090 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-28T16:06:01.631 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1053.ge58b9979.el9.noarch 13/13 2024-03-28T16:06:01.631 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1053.ge58b9979.el9.noarch 1/13 2024-03-28T16:06:01.631 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-28T16:06:01.631 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-28T16:06:01.631 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-28T16:06:01.631 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-28T16:06:01.632 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-28T16:06:01.632 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-28T16:06:01.632 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-28T16:06:01.632 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-28T16:06:01.632 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-28T16:06:01.632 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-28T16:06:01.632 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-28T16:06:02.001 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-28T16:06:02.001 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:02.001 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:06:02.001 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook-2:18.2.2-1053.ge58b9979.el9.noarch 2024-03-28T16:06:02.002 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T16:06:02.002 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T16:06:02.002 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T16:06:02.002 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T16:06:02.002 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T16:06:02.002 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T16:06:02.002 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T16:06:02.002 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T16:06:02.002 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T16:06:02.002 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T16:06:02.002 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T16:06:02.002 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T16:06:02.002 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:02.002 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:06:02.251 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-cephadm 2024-03-28T16:06:02.411 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-28T16:06:02.810 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:22 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:06:02.960 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:06:02.960 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-28T16:06:02.961 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-28T16:06:02.961 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-28T16:06:02.961 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:06:02.961 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1053.ge58b9979.el9 ceph-noarch 138 k 2024-03-28T16:06:02.961 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-28T16:06:02.961 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T16:06:02.961 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T16:06:02.961 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T16:06:02.961 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T16:06:02.961 INFO:teuthology.orchestra.run.smithi083.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T16:06:02.961 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:02.961 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:06:02.961 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-28T16:06:02.962 INFO:teuthology.orchestra.run.smithi083.stdout:Install 6 Packages 2024-03-28T16:06:02.962 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:02.962 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 6.8 M 2024-03-28T16:06:02.962 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 33 M 2024-03-28T16:06:02.962 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:06:03.406 INFO:teuthology.orchestra.run.smithi083.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-03-28T16:06:03.481 INFO:teuthology.orchestra.run.smithi083.stdout:(2/6): ceph-mgr-cephadm-18.2.2-1053.ge58b9979.e 540 kB/s | 138 kB 00:00 2024-03-28T16:06:03.482 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-28T16:06:03.516 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1053.ge58b9 12/12 2024-03-28T16:06:03.615 INFO:teuthology.orchestra.run.smithi083.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2024-03-28T16:06:03.732 INFO:teuthology.orchestra.run.smithi083.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 233 kB/s | 58 kB 00:00 2024-03-28T16:06:03.757 INFO:teuthology.orchestra.run.smithi083.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 360 kB/s | 51 kB 00:00 2024-03-28T16:06:03.782 INFO:teuthology.orchestra.run.smithi083.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.4 MB/s | 548 kB 00:00 2024-03-28T16:06:03.784 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:06:03.784 INFO:teuthology.orchestra.run.smithi083.stdout:Total 8.3 MB/s | 6.8 MB 00:00 2024-03-28T16:06:03.797 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:06:03.807 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:06:03.807 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:06:03.894 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:06:03.894 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:06:04.189 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:06:04.364 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-28T16:06:04.409 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-28T16:06:04.929 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-28T16:06:05.020 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-28T16:06:05.082 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-28T16:06:05.104 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1053.ge58b9979.el9.noarch 6/6 2024-03-28T16:06:05.829 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1053.ge58b9979.el9.noarch 6/6 2024-03-28T16:06:05.829 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1053.ge58b9979.el9.noarch 1/6 2024-03-28T16:06:05.829 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-28T16:06:05.829 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-28T16:06:05.829 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-28T16:06:05.829 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-28T16:06:06.157 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-28T16:06:06.158 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:06.158 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:06:06.158 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm-2:18.2.2-1053.ge58b9979.el9.noarch 2024-03-28T16:06:06.158 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T16:06:06.158 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T16:06:06.158 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T16:06:06.158 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T16:06:06.158 INFO:teuthology.orchestra.run.smithi083.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T16:06:06.158 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:06.158 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:06:06.176 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1053.ge58b9 12/12 2024-03-28T16:06:06.176 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1053.ge58b9 1/12 2024-03-28T16:06:06.177 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-28T16:06:06.177 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-28T16:06:06.177 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-28T16:06:06.177 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-28T16:06:06.177 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-28T16:06:06.177 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-28T16:06:06.177 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-28T16:06:06.177 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-28T16:06:06.177 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-28T16:06:06.177 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-28T16:06:06.399 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-fuse 2024-03-28T16:06:06.700 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-28T16:06:06.700 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:06.700 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:06:06.700 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1053.ge58b9979.el9.noarch 2024-03-28T16:06:06.700 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T16:06:06.700 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T16:06:06.700 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T16:06:06.700 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T16:06:06.701 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T16:06:06.701 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T16:06:06.701 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T16:06:06.701 INFO:teuthology.orchestra.run.smithi007.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T16:06:06.701 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T16:06:06.701 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T16:06:06.701 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T16:06:06.701 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:06.701 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:06:06.962 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:26 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:06:07.052 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-rook 2024-03-28T16:06:07.111 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:06:07.112 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:07.112 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-28T16:06:07.112 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:07.112 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:06:07.112 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 849 k 2024-03-28T16:06:07.112 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-28T16:06:07.112 INFO:teuthology.orchestra.run.smithi083.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T16:06:07.112 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:07.112 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:06:07.112 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:07.112 INFO:teuthology.orchestra.run.smithi083.stdout:Install 2 Packages 2024-03-28T16:06:07.112 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:07.112 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 928 k 2024-03-28T16:06:07.112 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 2.7 M 2024-03-28T16:06:07.113 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:06:07.251 INFO:teuthology.orchestra.run.smithi083.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 568 kB/s | 79 kB 00:00 2024-03-28T16:06:07.624 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:16 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:06:07.780 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:06:07.781 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-28T16:06:07.781 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-28T16:06:07.781 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-28T16:06:07.781 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:06:07.782 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook noarch 2:18.2.2-1053.ge58b9979.el9 ceph-noarch 49 k 2024-03-28T16:06:07.782 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-28T16:06:07.782 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T16:06:07.782 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T16:06:07.782 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T16:06:07.782 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T16:06:07.782 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T16:06:07.782 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T16:06:07.782 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T16:06:07.786 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T16:06:07.786 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T16:06:07.786 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T16:06:07.786 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T16:06:07.786 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T16:06:07.787 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:07.787 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:06:07.787 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-28T16:06:07.787 INFO:teuthology.orchestra.run.smithi007.stdout:Install 13 Packages 2024-03-28T16:06:07.787 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:07.787 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 2.0 M 2024-03-28T16:06:07.788 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 26 M 2024-03-28T16:06:07.788 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:06:07.901 INFO:teuthology.orchestra.run.smithi083.stdout:(2/2): ceph-fuse-18.2.2-1053.ge58b9979.el9.x86_ 1.1 MB/s | 849 kB 00:00 2024-03-28T16:06:07.902 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:06:07.903 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.1 MB/s | 928 kB 00:00 2024-03-28T16:06:07.903 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:06:07.924 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:06:07.924 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:06:07.966 INFO:teuthology.orchestra.run.smithi007.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 147 kB/s | 18 kB 00:00 2024-03-28T16:06:07.991 INFO:teuthology.orchestra.run.smithi007.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 164 kB/s | 24 kB 00:00 2024-03-28T16:06:07.998 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:06:07.998 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:06:08.041 INFO:teuthology.orchestra.run.smithi007.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-28T16:06:08.067 INFO:teuthology.orchestra.run.smithi007.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-28T16:06:08.092 INFO:teuthology.orchestra.run.smithi007.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-28T16:06:08.117 INFO:teuthology.orchestra.run.smithi007.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-03-28T16:06:08.194 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:06:08.276 INFO:teuthology.orchestra.run.smithi007.stdout:(7/13): ceph-mgr-rook-18.2.2-1053.ge58b9979.el9 114 kB/s | 49 kB 00:00 2024-03-28T16:06:08.325 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-28T16:06:08.368 INFO:teuthology.orchestra.run.smithi007.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-03-28T16:06:08.393 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-fuse-2:18.2.2-1053.ge58b9979.el9.x86_64 2/2 2024-03-28T16:06:08.393 INFO:teuthology.orchestra.run.smithi007.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2024-03-28T16:06:08.535 INFO:teuthology.orchestra.run.smithi007.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 783 kB/s | 203 kB 00:00 2024-03-28T16:06:08.560 INFO:teuthology.orchestra.run.smithi007.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 354 kB/s | 59 kB 00:00 2024-03-28T16:06:08.619 INFO:teuthology.orchestra.run.smithi007.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2024-03-28T16:06:09.074 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1053.ge58b9979.el9.x86_64 2/2 2024-03-28T16:06:09.074 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-fuse-2:18.2.2-1053.ge58b9979.el9.x86_64 1/2 2024-03-28T16:06:09.086 INFO:teuthology.orchestra.run.smithi007.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.4 MB/s | 1.0 MB 00:00 2024-03-28T16:06:09.088 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:06:09.088 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-28T16:06:09.089 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:06:09.104 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:06:09.105 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:06:09.204 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:06:09.206 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:06:09.360 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-28T16:06:09.361 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:09.361 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:06:09.361 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse-2:18.2.2-1053.ge58b9979.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-28T16:06:09.361 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:09.361 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:06:09.612 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-volume 2024-03-28T16:06:09.724 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:06:09.901 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-28T16:06:09.964 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-28T16:06:10.018 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-28T16:06:10.055 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-28T16:06:10.097 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-28T16:06:10.176 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:30 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:06:10.194 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-28T16:06:10.302 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-28T16:06:10.327 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:06:10.328 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:10.328 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-28T16:06:10.328 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:10.328 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:06:10.328 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume noarch 2:18.2.2-1053.ge58b9979.el9 ceph-noarch 264 k 2024-03-28T16:06:10.328 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-28T16:06:10.328 INFO:teuthology.orchestra.run.smithi083.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T16:06:10.328 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:10.328 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:06:10.329 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:10.329 INFO:teuthology.orchestra.run.smithi083.stdout:Install 2 Packages 2024-03-28T16:06:10.329 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:10.329 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 332 k 2024-03-28T16:06:10.329 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 1.5 M 2024-03-28T16:06:10.329 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:06:10.357 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-28T16:06:10.498 INFO:teuthology.orchestra.run.smithi083.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 408 kB/s | 69 kB 00:00 2024-03-28T16:06:10.865 INFO:teuthology.orchestra.run.smithi083.stdout:(2/2): ceph-volume-18.2.2-1053.ge58b9979.el9.no 493 kB/s | 264 kB 00:00 2024-03-28T16:06:10.866 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:06:10.866 INFO:teuthology.orchestra.run.smithi083.stdout:Total 618 kB/s | 332 kB 00:00 2024-03-28T16:06:10.866 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:06:10.874 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:06:10.874 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:06:10.913 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:06:10.913 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:06:11.054 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:06:11.137 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-28T16:06:11.178 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-28T16:06:11.213 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-28T16:06:11.237 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-28T16:06:11.261 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-28T16:06:11.278 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-volume-2:18.2.2-1053.ge58b9979.el9.noarch 2/2 2024-03-28T16:06:11.290 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-rook-2:18.2.2-1053.ge58b9979.el9.noarch 13/13 2024-03-28T16:06:11.293 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-volume-2:18.2.2-1053.ge58b9979.el9.noarch 2/2 2024-03-28T16:06:11.293 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T16:06:11.293 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T16:06:11.293 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:11.964 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-volume-2:18.2.2-1053.ge58b9979.el9.noarch 1/2 2024-03-28T16:06:12.087 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1053.ge58b9979.el9.noarch 13/13 2024-03-28T16:06:12.087 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1053.ge58b9979.el9.noarch 1/13 2024-03-28T16:06:12.087 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-28T16:06:12.087 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-28T16:06:12.087 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-28T16:06:12.087 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-28T16:06:12.088 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-28T16:06:12.088 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-28T16:06:12.088 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-28T16:06:12.088 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-28T16:06:12.088 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-28T16:06:12.088 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-28T16:06:12.088 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-28T16:06:12.306 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-28T16:06:12.306 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:12.306 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:06:12.306 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume-2:18.2.2-1053.ge58b9979.el9.noarch 2024-03-28T16:06:12.306 INFO:teuthology.orchestra.run.smithi083.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T16:06:12.307 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:12.307 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:06:12.445 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-28T16:06:12.445 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:12.446 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:06:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook-2:18.2.2-1053.ge58b9979.el9.noarch 2024-03-28T16:06:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T16:06:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T16:06:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T16:06:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T16:06:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T16:06:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T16:06:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T16:06:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T16:06:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T16:06:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T16:06:12.446 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T16:06:12.447 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T16:06:12.447 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:12.447 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:06:12.631 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados-devel 2024-03-28T16:06:12.701 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-cephadm 2024-03-28T16:06:13.208 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:33 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:06:13.272 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:22 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:06:13.355 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:06:13.355 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:13.355 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:06:13.356 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:13.356 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:06:13.356 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 127 k 2024-03-28T16:06:13.356 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:13.356 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:06:13.356 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:13.356 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:06:13.356 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:13.357 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 127 k 2024-03-28T16:06:13.357 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 456 k 2024-03-28T16:06:13.357 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:06:13.427 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:06:13.428 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-28T16:06:13.428 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-28T16:06:13.428 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-28T16:06:13.428 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:06:13.428 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1053.ge58b9979.el9 ceph-noarch 138 k 2024-03-28T16:06:13.429 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-28T16:06:13.429 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T16:06:13.429 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T16:06:13.429 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T16:06:13.429 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T16:06:13.429 INFO:teuthology.orchestra.run.smithi007.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T16:06:13.429 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:13.429 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:06:13.429 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-28T16:06:13.429 INFO:teuthology.orchestra.run.smithi007.stdout:Install 6 Packages 2024-03-28T16:06:13.429 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:13.430 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 6.8 M 2024-03-28T16:06:13.430 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 33 M 2024-03-28T16:06:13.430 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:06:13.581 INFO:teuthology.orchestra.run.smithi083.stdout:librados-devel-18.2.2-1053.ge58b9979.el9.x86_64 566 kB/s | 127 kB 00:00 2024-03-28T16:06:13.582 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:06:13.582 INFO:teuthology.orchestra.run.smithi083.stdout:Total 562 kB/s | 127 kB 00:00 2024-03-28T16:06:13.582 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:06:13.588 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:06:13.588 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:06:13.642 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:06:13.642 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:06:13.784 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:06:13.895 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librados-devel-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:14.064 INFO:teuthology.orchestra.run.smithi007.stdout:(1/6): ceph-mgr-cephadm-18.2.2-1053.ge58b9979.e 562 kB/s | 138 kB 00:00 2024-03-28T16:06:14.337 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados-devel-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:14.364 INFO:teuthology.orchestra.run.smithi007.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2024-03-28T16:06:14.656 INFO:teuthology.orchestra.run.smithi007.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 927 kB/s | 548 kB 00:00 2024-03-28T16:06:14.656 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados-devel-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:14.657 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:14.657 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:06:14.657 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:06:14.657 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:14.657 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:06:14.681 INFO:teuthology.orchestra.run.smithi007.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 182 kB/s | 58 kB 00:00 2024-03-28T16:06:14.750 INFO:teuthology.orchestra.run.smithi007.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 542 kB/s | 51 kB 00:00 2024-03-28T16:06:14.886 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install libcephfs2 2024-03-28T16:06:14.925 INFO:teuthology.orchestra.run.smithi007.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 224 kB/s | 248 kB 00:01 2024-03-28T16:06:14.928 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:06:14.928 INFO:teuthology.orchestra.run.smithi007.stdout:Total 4.5 MB/s | 6.8 MB 00:01 2024-03-28T16:06:14.943 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:06:14.955 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:06:14.955 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:06:15.053 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:06:15.053 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:06:15.350 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:06:15.450 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:35 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:06:15.534 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-28T16:06:15.549 INFO:teuthology.orchestra.run.smithi083.stdout:Package libcephfs2-2:18.2.2-1053.ge58b9979.el9.x86_64 is already installed. 2024-03-28T16:06:15.587 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-28T16:06:15.604 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:06:15.605 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-28T16:06:15.605 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:06:15.671 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install libcephfs-devel 2024-03-28T16:06:16.140 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-28T16:06:16.224 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-28T16:06:16.228 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:36 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:06:16.293 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-28T16:06:16.322 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1053.ge58b9979.el9.noarch 6/6 2024-03-28T16:06:16.377 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:06:16.378 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:16.378 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:06:16.378 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:16.378 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:06:16.378 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 31 k 2024-03-28T16:06:16.378 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:16.378 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:06:16.378 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:16.378 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:06:16.378 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:16.378 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 31 k 2024-03-28T16:06:16.379 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 139 k 2024-03-28T16:06:16.379 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:06:16.535 INFO:teuthology.orchestra.run.smithi083.stdout:libcephfs-devel-18.2.2-1053.ge58b9979.el9.x86_6 198 kB/s | 31 kB 00:00 2024-03-28T16:06:16.536 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:06:16.536 INFO:teuthology.orchestra.run.smithi083.stdout:Total 197 kB/s | 31 kB 00:00 2024-03-28T16:06:16.536 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:06:16.540 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:06:16.540 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:06:16.569 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:06:16.569 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:06:16.660 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:06:16.766 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs-devel-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:17.122 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1053.ge58b9979.el9.noarch 6/6 2024-03-28T16:06:17.122 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1053.ge58b9979.el9.noarch 1/6 2024-03-28T16:06:17.122 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-28T16:06:17.122 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-28T16:06:17.122 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-28T16:06:17.123 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-28T16:06:17.136 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:17.407 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs-devel-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:17.407 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:17.407 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:06:17.407 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:06:17.407 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:17.407 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:06:17.430 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-28T16:06:17.430 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:17.430 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:06:17.430 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm-2:18.2.2-1053.ge58b9979.el9.noarch 2024-03-28T16:06:17.430 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T16:06:17.430 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T16:06:17.430 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T16:06:17.430 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T16:06:17.431 INFO:teuthology.orchestra.run.smithi007.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T16:06:17.431 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:17.431 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:06:17.578 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados2 2024-03-28T16:06:17.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-fuse 2024-03-28T16:06:18.142 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:38 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:06:18.239 INFO:teuthology.orchestra.run.smithi083.stdout:Package librados2-2:18.2.2-1053.ge58b9979.el9.x86_64 is already installed. 2024-03-28T16:06:18.283 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:27 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:06:18.291 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:06:18.292 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-28T16:06:18.292 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:06:18.347 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librbd1 2024-03-28T16:06:18.439 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:06:18.439 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:18.439 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-28T16:06:18.439 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:18.439 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:06:18.439 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 849 k 2024-03-28T16:06:18.440 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-28T16:06:18.440 INFO:teuthology.orchestra.run.smithi007.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T16:06:18.440 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:18.440 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:06:18.440 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:18.440 INFO:teuthology.orchestra.run.smithi007.stdout:Install 2 Packages 2024-03-28T16:06:18.440 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:18.441 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 928 k 2024-03-28T16:06:18.441 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 2.7 M 2024-03-28T16:06:18.441 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:06:18.578 INFO:teuthology.orchestra.run.smithi007.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 574 kB/s | 79 kB 00:00 2024-03-28T16:06:18.753 INFO:teuthology.orchestra.run.smithi007.stdout:(2/2): ceph-fuse-18.2.2-1053.ge58b9979.el9.x86_ 2.7 MB/s | 849 kB 00:00 2024-03-28T16:06:18.755 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:06:18.755 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.9 MB/s | 928 kB 00:00 2024-03-28T16:06:18.755 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:06:18.777 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:06:18.777 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:06:18.854 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:06:18.854 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:06:18.912 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:38 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:06:19.008 INFO:teuthology.orchestra.run.smithi083.stdout:Package librbd1-2:18.2.2-1053.ge58b9979.el9.x86_64 is already installed. 2024-03-28T16:06:19.049 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:06:19.060 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:06:19.062 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-28T16:06:19.062 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:06:19.129 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rados 2024-03-28T16:06:19.197 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-28T16:06:19.272 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-fuse-2:18.2.2-1053.ge58b9979.el9.x86_64 2/2 2024-03-28T16:06:19.686 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:39 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:06:19.782 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rados-2:18.2.2-1053.ge58b9979.el9.x86_64 is already installed. 2024-03-28T16:06:19.834 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:06:19.835 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-28T16:06:19.835 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:06:19.902 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rgw 2024-03-28T16:06:19.927 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1053.ge58b9979.el9.x86_64 2/2 2024-03-28T16:06:19.927 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-fuse-2:18.2.2-1053.ge58b9979.el9.x86_64 1/2 2024-03-28T16:06:20.222 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-28T16:06:20.222 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:20.223 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:06:20.223 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-2:18.2.2-1053.ge58b9979.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-28T16:06:20.223 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:20.223 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:06:20.460 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:40 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:06:20.495 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-volume 2024-03-28T16:06:20.556 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rgw-2:18.2.2-1053.ge58b9979.el9.x86_64 is already installed. 2024-03-28T16:06:20.609 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:06:20.610 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-28T16:06:20.610 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:06:20.675 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-cephfs 2024-03-28T16:06:21.068 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:30 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:06:21.225 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:06:21.225 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:21.225 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-28T16:06:21.225 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:21.225 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:06:21.226 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume noarch 2:18.2.2-1053.ge58b9979.el9 ceph-noarch 264 k 2024-03-28T16:06:21.226 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-28T16:06:21.226 INFO:teuthology.orchestra.run.smithi007.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T16:06:21.226 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:21.226 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:06:21.226 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:21.226 INFO:teuthology.orchestra.run.smithi007.stdout:Install 2 Packages 2024-03-28T16:06:21.226 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:21.227 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 332 k 2024-03-28T16:06:21.227 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.5 M 2024-03-28T16:06:21.227 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:06:21.235 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:41 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:06:21.331 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-cephfs-2:18.2.2-1053.ge58b9979.el9.x86_64 is already installed. 2024-03-28T16:06:21.350 INFO:teuthology.orchestra.run.smithi007.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 560 kB/s | 69 kB 00:00 2024-03-28T16:06:21.384 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:06:21.385 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-28T16:06:21.385 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:06:21.453 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rbd 2024-03-28T16:06:21.592 INFO:teuthology.orchestra.run.smithi007.stdout:(2/2): ceph-volume-18.2.2-1053.ge58b9979.el9.no 723 kB/s | 264 kB 00:00 2024-03-28T16:06:21.593 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:06:21.594 INFO:teuthology.orchestra.run.smithi007.stdout:Total 905 kB/s | 332 kB 00:00 2024-03-28T16:06:21.594 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:06:21.603 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:06:21.603 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:06:21.640 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:06:21.640 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:06:21.788 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:06:21.973 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-28T16:06:22.009 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:42 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:06:22.031 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-volume-2:18.2.2-1053.ge58b9979.el9.noarch 2/2 2024-03-28T16:06:22.031 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-volume-2:18.2.2-1053.ge58b9979.el9.noarch 2/2 2024-03-28T16:06:22.031 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T16:06:22.031 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T16:06:22.032 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:22.105 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rbd-2:18.2.2-1053.ge58b9979.el9.x86_64 is already installed. 2024-03-28T16:06:22.157 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:06:22.158 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-28T16:06:22.158 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:06:22.224 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-fuse 2024-03-28T16:06:22.723 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-volume-2:18.2.2-1053.ge58b9979.el9.noarch 1/2 2024-03-28T16:06:22.786 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:42 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:06:22.936 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:06:22.936 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:22.937 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-03-28T16:06:22.937 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:22.937 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:06:22.937 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 87 k 2024-03-28T16:06:22.937 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:22.937 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:06:22.937 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:22.938 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:06:22.938 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:22.938 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 87 k 2024-03-28T16:06:22.938 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 227 k 2024-03-28T16:06:22.938 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:06:23.070 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-28T16:06:23.070 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:23.071 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:06:23.071 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume-2:18.2.2-1053.ge58b9979.el9.noarch 2024-03-28T16:06:23.071 INFO:teuthology.orchestra.run.smithi007.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T16:06:23.071 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:23.071 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:06:23.263 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-fuse-18.2.2-1053.ge58b9979.el9.x86_64.rpm 266 kB/s | 87 kB 00:00 2024-03-28T16:06:23.264 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:06:23.264 INFO:teuthology.orchestra.run.smithi083.stdout:Total 265 kB/s | 87 kB 00:00 2024-03-28T16:06:23.264 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:06:23.271 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:06:23.271 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:06:23.326 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:06:23.327 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:06:23.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados-devel 2024-03-28T16:06:23.463 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:06:23.570 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-fuse-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:23.965 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:24.010 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:33 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:06:24.164 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:06:24.165 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:24.165 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:06:24.165 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:24.165 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:06:24.165 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 127 k 2024-03-28T16:06:24.165 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:24.165 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:06:24.165 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:24.165 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:06:24.165 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:24.166 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 127 k 2024-03-28T16:06:24.166 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 456 k 2024-03-28T16:06:24.166 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:06:24.251 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-fuse-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:24.251 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:24.251 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:06:24.251 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:06:24.251 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:24.251 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:06:24.374 INFO:teuthology.orchestra.run.smithi007.stdout:librados-devel-18.2.2-1053.ge58b9979.el9.x86_64 612 kB/s | 127 kB 00:00 2024-03-28T16:06:24.374 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:06:24.374 INFO:teuthology.orchestra.run.smithi007.stdout:Total 607 kB/s | 127 kB 00:00 2024-03-28T16:06:24.374 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:06:24.381 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:06:24.381 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:06:24.437 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:06:24.438 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:06:24.514 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-mirror 2024-03-28T16:06:24.585 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:06:24.700 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados-devel-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:25.094 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:45 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:06:25.143 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados-devel-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:25.248 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:06:25.249 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:25.249 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:06:25.249 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:25.249 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:06:25.249 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 3.0 M 2024-03-28T16:06:25.249 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:25.249 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:06:25.249 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:25.249 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:06:25.249 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:25.249 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 3.0 M 2024-03-28T16:06:25.250 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 12 M 2024-03-28T16:06:25.250 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:06:25.444 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados-devel-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:25.444 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:25.444 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:06:25.444 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:06:25.444 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:25.444 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:06:25.626 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-mirror-18.2.2-1053.ge58b9979.el9.x86_64.rpm 7.9 MB/s | 3.0 MB 00:00 2024-03-28T16:06:25.627 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:06:25.627 INFO:teuthology.orchestra.run.smithi083.stdout:Total 7.9 MB/s | 3.0 MB 00:00 2024-03-28T16:06:25.627 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:06:25.636 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:06:25.636 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:06:25.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs2 2024-03-28T16:06:25.708 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:06:25.709 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:06:26.123 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:06:26.194 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-mirror-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:26.219 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:26.219 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T16:06:26.219 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T16:06:26.219 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T16:06:26.219 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T16:06:26.219 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:26.259 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:35 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:06:26.359 INFO:teuthology.orchestra.run.smithi007.stdout:Package libcephfs2-2:18.2.2-1053.ge58b9979.el9.x86_64 is already installed. 2024-03-28T16:06:26.414 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:06:26.415 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-28T16:06:26.415 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:06:26.487 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs-devel 2024-03-28T16:06:27.054 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:36 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:06:27.208 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:06:27.209 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:27.209 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:06:27.209 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:27.209 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:06:27.209 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 31 k 2024-03-28T16:06:27.209 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:27.209 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:06:27.209 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:27.209 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:06:27.209 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:27.210 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 31 k 2024-03-28T16:06:27.210 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 139 k 2024-03-28T16:06:27.210 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:06:27.304 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-mirror-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:27.304 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:27.304 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:06:27.304 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:06:27.304 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:27.304 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:06:27.570 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-nbd 2024-03-28T16:06:27.628 INFO:teuthology.orchestra.run.smithi007.stdout:libcephfs-devel-18.2.2-1053.ge58b9979.el9.x86_6 74 kB/s | 31 kB 00:00 2024-03-28T16:06:27.628 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:06:27.628 INFO:teuthology.orchestra.run.smithi007.stdout:Total 74 kB/s | 31 kB 00:00 2024-03-28T16:06:27.629 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:06:27.632 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:06:27.632 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:06:27.659 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:06:27.659 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:06:27.757 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:06:27.904 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs-devel-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:28.143 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:48 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:06:28.286 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:28.292 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:06:28.292 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:28.293 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-03-28T16:06:28.293 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:28.293 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:06:28.293 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 172 k 2024-03-28T16:06:28.293 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:28.293 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:06:28.293 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:28.293 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:06:28.293 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:28.293 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 172 k 2024-03-28T16:06:28.293 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 494 k 2024-03-28T16:06:28.293 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:06:28.553 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs-devel-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:28.553 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:28.553 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:06:28.553 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:06:28.553 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:28.554 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:06:28.663 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-nbd-18.2.2-1053.ge58b9979.el9.x86_64.rpm 465 kB/s | 172 kB 00:00 2024-03-28T16:06:28.663 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:06:28.664 INFO:teuthology.orchestra.run.smithi083.stdout:Total 463 kB/s | 172 kB 00:00 2024-03-28T16:06:28.664 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:06:28.672 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:06:28.672 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:06:28.728 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:06:28.728 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:06:28.746 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados2 2024-03-28T16:06:28.871 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:06:28.970 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-nbd-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:29.318 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:38 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:06:29.416 INFO:teuthology.orchestra.run.smithi007.stdout:Package librados2-2:18.2.2-1053.ge58b9979.el9.x86_64 is already installed. 2024-03-28T16:06:29.456 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:29.472 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:06:29.473 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-28T16:06:29.473 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:06:29.553 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librbd1 2024-03-28T16:06:29.762 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-nbd-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:29.762 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:29.762 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:06:29.763 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:06:29.763 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:29.763 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:06:30.013 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-base-debuginfo 2024-03-28T16:06:30.128 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:39 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:06:30.228 INFO:teuthology.orchestra.run.smithi007.stdout:Package librbd1-2:18.2.2-1053.ge58b9979.el9.x86_64 is already installed. 2024-03-28T16:06:30.283 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:06:30.284 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-28T16:06:30.284 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:06:30.355 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rados 2024-03-28T16:06:30.576 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:50 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:06:30.729 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:06:30.730 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:30.730 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:06:30.730 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:30.730 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:06:30.730 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 67 M 2024-03-28T16:06:30.730 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-28T16:06:30.730 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 20 M 2024-03-28T16:06:30.730 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:30.731 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:06:30.731 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:30.731 INFO:teuthology.orchestra.run.smithi083.stdout:Install 2 Packages 2024-03-28T16:06:30.731 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:30.731 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 87 M 2024-03-28T16:06:30.731 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 482 M 2024-03-28T16:06:30.731 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:06:30.928 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:39 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:06:31.028 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rados-2:18.2.2-1053.ge58b9979.el9.x86_64 is already installed. 2024-03-28T16:06:31.083 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:06:31.084 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-28T16:06:31.085 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:06:31.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rgw 2024-03-28T16:06:31.731 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:40 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:06:31.829 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rgw-2:18.2.2-1053.ge58b9979.el9.x86_64 is already installed. 2024-03-28T16:06:31.885 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:06:31.886 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-28T16:06:31.886 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:06:31.961 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-cephfs 2024-03-28T16:06:32.532 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:41 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:06:32.631 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-cephfs-2:18.2.2-1053.ge58b9979.el9.x86_64 is already installed. 2024-03-28T16:06:32.687 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:06:32.688 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-28T16:06:32.688 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:06:32.764 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rbd 2024-03-28T16:06:33.333 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:42 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:06:33.434 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rbd-2:18.2.2-1053.ge58b9979.el9.x86_64 is already installed. 2024-03-28T16:06:33.489 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:06:33.490 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-28T16:06:33.490 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:06:33.561 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-fuse 2024-03-28T16:06:34.134 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:43 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:06:34.242 INFO:teuthology.orchestra.run.smithi083.stdout:(1/2): ceph-base-debuginfo-18.2.2-1053.ge58b997 19 MB/s | 67 MB 00:03 2024-03-28T16:06:34.289 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:06:34.290 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:34.290 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-03-28T16:06:34.290 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:34.290 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:06:34.290 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 87 k 2024-03-28T16:06:34.290 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:34.290 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:06:34.290 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:34.290 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:06:34.291 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:34.291 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 87 k 2024-03-28T16:06:34.291 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 227 k 2024-03-28T16:06:34.291 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:06:34.606 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-fuse-18.2.2-1053.ge58b9979.el9.x86_64.rpm 275 kB/s | 87 kB 00:00 2024-03-28T16:06:34.606 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:06:34.607 INFO:teuthology.orchestra.run.smithi007.stdout:Total 274 kB/s | 87 kB 00:00 2024-03-28T16:06:34.607 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:06:34.614 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:06:34.614 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:06:34.672 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:06:34.672 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:06:34.819 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:06:34.923 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-fuse-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:35.307 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:35.592 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-fuse-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:35.592 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:35.592 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:06:35.592 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:06:35.592 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:35.592 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:06:35.842 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-mirror 2024-03-28T16:06:36.412 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:45 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:06:36.569 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:06:36.569 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:36.570 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:06:36.570 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:36.570 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:06:36.570 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 3.0 M 2024-03-28T16:06:36.570 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:36.570 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:06:36.570 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:36.570 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:06:36.570 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:36.570 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 3.0 M 2024-03-28T16:06:36.570 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 12 M 2024-03-28T16:06:36.570 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:06:37.801 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-mirror-18.2.2-1053.ge58b9979.el9.x86_64.rpm 2.4 MB/s | 3.0 MB 00:01 2024-03-28T16:06:37.802 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:06:37.802 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.4 MB/s | 3.0 MB 00:01 2024-03-28T16:06:37.803 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:06:37.813 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:06:37.813 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:06:37.889 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:06:37.890 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:06:38.326 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:06:38.394 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-mirror-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:38.418 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:38.418 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T16:06:38.418 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T16:06:38.419 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T16:06:38.419 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T16:06:38.419 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:39.184 INFO:teuthology.orchestra.run.smithi083.stdout:(2/2): ceph-debuginfo-18.2.2-1053.ge58b9979.el9 2.4 MB/s | 20 MB 00:08 2024-03-28T16:06:39.185 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:06:39.186 INFO:teuthology.orchestra.run.smithi083.stdout:Total 10 MB/s | 87 MB 00:08 2024-03-28T16:06:39.186 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:06:39.191 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:06:39.191 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:06:39.576 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-mirror-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:39.576 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:39.577 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:06:39.577 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:06:39.577 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:39.577 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:06:39.755 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:06:39.755 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:06:39.848 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-nbd 2024-03-28T16:06:40.427 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:49 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:06:40.582 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:06:40.583 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:40.583 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-03-28T16:06:40.583 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:40.583 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:06:40.583 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 172 k 2024-03-28T16:06:40.584 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:40.584 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:06:40.584 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:40.584 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:06:40.584 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:40.584 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 172 k 2024-03-28T16:06:40.584 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 494 k 2024-03-28T16:06:40.584 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:06:40.938 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-nbd-18.2.2-1053.ge58b9979.el9.x86_64.rpm 485 kB/s | 172 kB 00:00 2024-03-28T16:06:40.939 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:06:40.939 INFO:teuthology.orchestra.run.smithi007.stdout:Total 484 kB/s | 172 kB 00:00 2024-03-28T16:06:40.939 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:06:40.948 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:06:40.948 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:06:41.006 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:06:41.006 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:06:41.168 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:06:41.292 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-nbd-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:41.742 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:42.034 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-nbd-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:06:42.034 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:42.035 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:06:42.035 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:06:42.035 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:42.035 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:06:42.287 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-base-debuginfo 2024-03-28T16:06:42.610 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:06:42.877 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:51 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:06:43.036 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:06:43.036 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:43.036 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:06:43.036 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:43.036 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:06:43.036 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 67 M 2024-03-28T16:06:43.036 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-28T16:06:43.036 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 20 M 2024-03-28T16:06:43.037 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:43.037 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:06:43.037 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:06:43.037 INFO:teuthology.orchestra.run.smithi007.stdout:Install 2 Packages 2024-03-28T16:06:43.037 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:06:43.037 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 87 M 2024-03-28T16:06:43.037 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 482 M 2024-03-28T16:06:43.038 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:06:45.009 INFO:teuthology.orchestra.run.smithi007.stdout:(1/2): ceph-debuginfo-18.2.2-1053.ge58b9979.el9 10 MB/s | 20 MB 00:01 2024-03-28T16:06:47.276 INFO:teuthology.orchestra.run.smithi007.stdout:(2/2): ceph-base-debuginfo-18.2.2-1053.ge58b997 16 MB/s | 67 MB 00:04 2024-03-28T16:06:47.278 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:06:47.278 INFO:teuthology.orchestra.run.smithi007.stdout:Total 21 MB/s | 87 MB 00:04 2024-03-28T16:06:47.279 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:06:47.284 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:06:47.284 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:06:47.867 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:06:47.868 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:06:49.606 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 1/2 2024-03-28T16:06:49.693 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-base-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 2/2 2024-03-28T16:06:50.884 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:06:53.088 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-base-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 2/2 2024-03-28T16:06:53.088 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-base-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 1/2 2024-03-28T16:06:53.351 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2/2 2024-03-28T16:06:53.352 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:53.352 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:06:53.352 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:06:53.352 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:06:53.352 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:53.352 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:06:53.605 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-common-debuginfo 2024-03-28T16:06:54.154 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:14 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:06:54.304 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:06:54.304 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:54.304 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:06:54.304 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:54.305 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:06:54.305 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 399 M 2024-03-28T16:06:54.305 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:54.305 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:06:54.305 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:06:54.305 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:06:54.305 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:06:54.305 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 399 M 2024-03-28T16:06:54.305 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 1.6 G 2024-03-28T16:06:54.306 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:06:58.289 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 1/2 2024-03-28T16:06:58.363 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-base-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 2/2 2024-03-28T16:07:01.754 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-base-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 2/2 2024-03-28T16:07:01.755 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-base-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 1/2 2024-03-28T16:07:02.020 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2/2 2024-03-28T16:07:02.020 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:07:02.020 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:07:02.020 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:07:02.020 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:07:02.020 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:07:02.020 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:07:02.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-common-debuginfo 2024-03-28T16:07:02.847 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:11 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:07:03.002 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:07:03.003 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:07:03.003 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:07:03.003 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:07:03.003 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:07:03.003 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 399 M 2024-03-28T16:07:03.003 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:07:03.003 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:07:03.003 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:07:03.004 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:07:03.004 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:07:03.004 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 399 M 2024-03-28T16:07:03.004 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.6 G 2024-03-28T16:07:03.004 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:07:22.795 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-common-debuginfo-18.2.2-1053.ge58b9979.el9 14 MB/s | 399 MB 00:28 2024-03-28T16:07:22.795 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:07:22.796 INFO:teuthology.orchestra.run.smithi083.stdout:Total 14 MB/s | 399 MB 00:28 2024-03-28T16:07:22.796 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:07:22.800 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:07:22.800 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:07:25.177 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:07:25.177 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:07:45.676 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-common-debuginfo-18.2.2-1053.ge58b9979.el9 9.4 MB/s | 399 MB 00:42 2024-03-28T16:07:45.677 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:07:45.678 INFO:teuthology.orchestra.run.smithi007.stdout:Total 9.4 MB/s | 399 MB 00:42 2024-03-28T16:07:45.678 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:07:45.682 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:07:45.682 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:07:48.166 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:07:48.167 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:08:05.663 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:08:05.755 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-common-debuginfo-2:18.2.2-1053.ge58b9979.el9.x8 1/1 2024-03-28T16:08:14.473 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-debuginfo-2:18.2.2-1053.ge58b9979.el9.x8 1/1 2024-03-28T16:08:14.730 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-common-debuginfo-2:18.2.2-1053.ge58b9979.el9.x8 1/1 2024-03-28T16:08:14.733 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:08:14.733 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:08:14.734 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:08:14.734 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:08:14.734 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:08:14.945 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-28T16:08:15.527 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:03:35 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:08:15.686 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:08:15.687 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-28T16:08:15.687 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:08:15.687 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-28T16:08:15.687 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:08:15.687 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 1.4 M 2024-03-28T16:08:15.687 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:08:15.687 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:08:15.688 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-28T16:08:15.688 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:08:15.688 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:08:15.688 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 1.4 M 2024-03-28T16:08:15.688 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 6.2 M 2024-03-28T16:08:15.688 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:08:16.969 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-immutable-object-cache-debuginfo-18.2.2-10 1.1 MB/s | 1.4 MB 00:01 2024-03-28T16:08:16.970 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:08:16.970 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.1 MB/s | 1.4 MB 00:01 2024-03-28T16:08:16.970 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:08:16.974 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:08:16.977 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:08:17.015 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:08:17.015 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:08:17.262 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:08:17.368 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.2.2-1053. 1/1 2024-03-28T16:08:17.805 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.2.2-1053. 1/1 2024-03-28T16:08:18.064 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.2.2-1053. 1/1 2024-03-28T16:08:18.065 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:08:18.065 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:08:18.065 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:08:18.065 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:08:18.065 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:08:18.236 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-28T16:08:18.792 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:03:38 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:08:18.942 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:08:18.943 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:08:18.943 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:08:18.943 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:08:18.943 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:08:18.943 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 162 M 2024-03-28T16:08:18.943 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:08:18.943 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:08:18.943 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:08:18.943 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:08:18.944 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:08:18.944 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 162 M 2024-03-28T16:08:18.944 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 679 M 2024-03-28T16:08:18.944 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:08:30.850 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:08:30.949 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-common-debuginfo-2:18.2.2-1053.ge58b9979.el9.x8 1/1 2024-03-28T16:08:38.790 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-debuginfo-2:18.2.2-1053.ge58b9979.el9.x8 1/1 2024-03-28T16:08:39.068 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-common-debuginfo-2:18.2.2-1053.ge58b9979.el9.x8 1/1 2024-03-28T16:08:39.068 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:08:39.068 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:08:39.068 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:08:39.069 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:08:39.069 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:08:39.413 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-28T16:08:39.995 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:48 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:08:40.157 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:08:40.157 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-28T16:08:40.157 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:08:40.158 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-28T16:08:40.158 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:08:40.158 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 1.4 M 2024-03-28T16:08:40.158 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:08:40.158 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:08:40.158 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-28T16:08:40.158 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:08:40.158 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:08:40.158 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 1.4 M 2024-03-28T16:08:40.158 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 6.2 M 2024-03-28T16:08:40.158 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:08:40.608 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-immutable-object-cache-debuginfo-18.2.2-10 3.1 MB/s | 1.4 MB 00:00 2024-03-28T16:08:40.608 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:08:40.609 INFO:teuthology.orchestra.run.smithi007.stdout:Total 3.1 MB/s | 1.4 MB 00:00 2024-03-28T16:08:40.609 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:08:40.612 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:08:40.612 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:08:40.655 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:08:40.656 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:08:40.921 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:08:41.025 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.2.2-1053. 1/1 2024-03-28T16:08:41.420 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.2.2-1053. 1/1 2024-03-28T16:08:41.683 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.2.2-1053. 1/1 2024-03-28T16:08:41.683 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:08:41.683 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:08:41.683 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:08:41.683 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:08:41.683 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:08:41.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-28T16:08:42.440 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:51 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:08:42.600 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:08:42.601 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:08:42.601 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:08:42.601 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:08:42.601 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:08:42.602 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 162 M 2024-03-28T16:08:42.602 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:08:42.602 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:08:42.602 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:08:42.602 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:08:42.602 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:08:42.602 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 162 M 2024-03-28T16:08:42.602 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 679 M 2024-03-28T16:08:42.602 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:08:44.812 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-radosgw-debuginfo-18.2.2-1053.ge58b9979.el 6.3 MB/s | 162 MB 00:25 2024-03-28T16:08:44.812 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:08:44.813 INFO:teuthology.orchestra.run.smithi083.stdout:Total 6.3 MB/s | 162 MB 00:25 2024-03-28T16:08:44.813 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:08:44.817 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:08:44.817 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:08:45.801 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:08:45.801 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:08:51.294 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-radosgw-debuginfo-18.2.2-1053.ge58b9979.el 19 MB/s | 162 MB 00:08 2024-03-28T16:08:51.294 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:08:51.295 INFO:teuthology.orchestra.run.smithi007.stdout:Total 19 MB/s | 162 MB 00:08 2024-03-28T16:08:51.295 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:08:51.299 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:08:51.299 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:08:52.329 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:08:52.329 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:09:02.342 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:09:02.508 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-radosgw-debuginfo-2:18.2.2-1053.ge58b9979.el9.x 1/1 2024-03-28T16:09:06.807 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.2.2-1053.ge58b9979.el9.x 1/1 2024-03-28T16:09:07.074 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-radosgw-debuginfo-2:18.2.2-1053.ge58b9979.el9.x 1/1 2024-03-28T16:09:07.074 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:09:07.074 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:09:07.075 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:09:07.075 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:09:07.075 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:09:07.276 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-test-debuginfo 2024-03-28T16:09:07.832 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:04:27 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:09:07.980 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:09:07.981 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:09:07.981 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:09:07.981 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:09:07.981 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:09:07.981 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 582 M 2024-03-28T16:09:07.981 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:09:07.982 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:09:07.982 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:09:07.982 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:09:07.982 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:09:07.982 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 582 M 2024-03-28T16:09:07.982 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 2.6 G 2024-03-28T16:09:07.982 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:09:09.852 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:09:09.951 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-radosgw-debuginfo-2:18.2.2-1053.ge58b9979.el9.x 1/1 2024-03-28T16:09:14.310 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.2.2-1053.ge58b9979.el9.x 1/1 2024-03-28T16:09:14.583 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-radosgw-debuginfo-2:18.2.2-1053.ge58b9979.el9.x 1/1 2024-03-28T16:09:14.584 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:09:14.584 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:09:14.584 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:09:14.584 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:09:14.584 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:09:14.790 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-test-debuginfo 2024-03-28T16:09:15.360 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:24 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:09:15.516 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:09:15.516 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:09:15.517 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:09:15.517 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:09:15.517 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:09:15.517 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 582 M 2024-03-28T16:09:15.517 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:09:15.517 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:09:15.517 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:09:15.517 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:09:15.518 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:09:15.518 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 582 M 2024-03-28T16:09:15.518 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 2.6 G 2024-03-28T16:09:15.518 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:09:54.015 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-test-debuginfo-18.2.2-1053.ge58b9979.el9.x 15 MB/s | 582 MB 00:38 2024-03-28T16:09:54.016 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:09:54.016 INFO:teuthology.orchestra.run.smithi007.stdout:Total 15 MB/s | 582 MB 00:38 2024-03-28T16:09:54.016 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:09:54.020 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:09:54.020 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:09:54.683 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-test-debuginfo-18.2.2-1053.ge58b9979.el9.x 12 MB/s | 582 MB 00:46 2024-03-28T16:09:54.684 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:09:54.684 INFO:teuthology.orchestra.run.smithi083.stdout:Total 12 MB/s | 582 MB 00:46 2024-03-28T16:09:54.685 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:09:54.690 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:09:54.690 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:09:57.635 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:09:57.635 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:09:58.126 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:09:58.126 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:10:59.095 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:10:59.218 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-test-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 1/1 2024-03-28T16:11:02.233 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:11:02.333 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-test-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 1/1 2024-03-28T16:11:06.241 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-test-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 1/1 2024-03-28T16:11:06.507 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-test-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 1/1 2024-03-28T16:11:06.508 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:11:06.508 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:11:06.508 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:11:06.508 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:11:06.508 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:11:06.760 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-base-debuginfo 2024-03-28T16:11:07.322 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:27 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:11:07.419 INFO:teuthology.orchestra.run.smithi083.stdout:Package ceph-base-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 is already installed. 2024-03-28T16:11:07.470 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:11:07.471 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-28T16:11:07.471 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:11:07.539 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-debuginfo 2024-03-28T16:11:08.103 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:28 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:11:08.251 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:11:08.252 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:11:08.252 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:11:08.252 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:11:08.252 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:11:08.253 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 19 M 2024-03-28T16:11:08.253 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:11:08.253 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:11:08.253 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:11:08.253 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:11:08.253 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:11:08.253 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 19 M 2024-03-28T16:11:08.253 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 101 M 2024-03-28T16:11:08.253 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:11:09.505 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-mgr-debuginfo-18.2.2-1053.ge58b9979.el9.x8 15 MB/s | 19 MB 00:01 2024-03-28T16:11:09.505 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:11:09.506 INFO:teuthology.orchestra.run.smithi083.stdout:Total 15 MB/s | 19 MB 00:01 2024-03-28T16:11:09.506 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:11:09.510 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:11:09.510 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:11:09.655 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:11:09.656 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:11:11.150 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-test-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 1/1 2024-03-28T16:11:11.413 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-test-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 1/1 2024-03-28T16:11:11.413 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:11:11.414 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:11:11.414 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:11:11.414 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:11:11.414 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:11:11.687 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-base-debuginfo 2024-03-28T16:11:11.858 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:11:12.053 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:11:12.261 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:21 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:11:12.360 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-base-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 is already installed. 2024-03-28T16:11:12.415 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:11:12.416 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-28T16:11:12.416 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:11:12.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-debuginfo 2024-03-28T16:11:13.054 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:22 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:11:13.078 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:11:13.207 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:11:13.207 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:11:13.208 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:11:13.208 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:11:13.208 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:11:13.208 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 19 M 2024-03-28T16:11:13.208 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:11:13.208 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:11:13.208 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:11:13.208 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:11:13.208 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:11:13.208 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 19 M 2024-03-28T16:11:13.209 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 101 M 2024-03-28T16:11:13.209 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:11:13.330 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:11:13.331 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:11:13.331 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:11:13.331 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:11:13.331 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:11:13.331 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:11:13.518 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mds-debuginfo 2024-03-28T16:11:14.079 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:34 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:11:14.227 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:11:14.227 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:11:14.227 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:11:14.227 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:11:14.227 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:11:14.227 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 47 M 2024-03-28T16:11:14.228 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:11:14.228 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:11:14.228 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:11:14.228 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:11:14.228 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:11:14.228 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 47 M 2024-03-28T16:11:14.228 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 171 M 2024-03-28T16:11:14.229 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:11:14.659 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-mgr-debuginfo-18.2.2-1053.ge58b9979.el9.x8 13 MB/s | 19 MB 00:01 2024-03-28T16:11:14.660 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:11:14.660 INFO:teuthology.orchestra.run.smithi007.stdout:Total 13 MB/s | 19 MB 00:01 2024-03-28T16:11:14.661 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:11:14.665 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:11:14.665 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:11:14.819 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:11:14.819 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:11:16.949 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-mds-debuginfo-18.2.2-1053.ge58b9979.el9.x8 17 MB/s | 47 MB 00:02 2024-03-28T16:11:16.950 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:11:16.950 INFO:teuthology.orchestra.run.smithi083.stdout:Total 17 MB/s | 47 MB 00:02 2024-03-28T16:11:16.951 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:11:16.955 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:11:16.955 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:11:17.120 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:11:17.218 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:11:17.264 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:11:17.265 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:11:18.226 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:11:18.495 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:11:18.495 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:11:18.495 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:11:18.495 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:11:18.495 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:11:18.495 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:11:18.680 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mds-debuginfo 2024-03-28T16:11:19.252 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:28 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:11:19.406 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:11:19.406 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:11:19.406 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:11:19.407 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:11:19.407 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:11:19.407 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 47 M 2024-03-28T16:11:19.407 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:11:19.407 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:11:19.407 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:11:19.407 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:11:19.407 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:11:19.407 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 47 M 2024-03-28T16:11:19.407 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 171 M 2024-03-28T16:11:19.407 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:11:21.870 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-mds-debuginfo-18.2.2-1053.ge58b9979.el9.x8 19 MB/s | 47 MB 00:02 2024-03-28T16:11:21.871 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:11:21.871 INFO:teuthology.orchestra.run.smithi007.stdout:Total 19 MB/s | 47 MB 00:02 2024-03-28T16:11:21.872 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:11:21.876 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:11:21.876 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:11:21.955 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:11:22.122 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mds-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:11:22.201 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:11:22.201 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:11:23.488 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:11:23.764 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mds-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:11:23.764 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:11:23.764 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:11:23.764 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:11:23.764 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:11:23.764 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:11:23.958 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mon-debuginfo 2024-03-28T16:11:24.516 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:44 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:11:24.663 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:11:24.664 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:11:24.664 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:11:24.664 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:11:24.664 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:11:24.664 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 98 M 2024-03-28T16:11:24.664 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:11:24.664 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:11:24.665 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:11:24.665 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:11:24.665 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:11:24.665 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 98 M 2024-03-28T16:11:24.665 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 450 M 2024-03-28T16:11:24.665 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:11:27.155 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:11:27.245 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mds-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:11:28.662 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:11:28.926 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mds-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:11:28.927 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:11:28.927 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:11:28.927 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:11:28.927 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:11:28.927 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:11:29.111 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mon-debuginfo 2024-03-28T16:11:29.677 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:38 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:11:29.831 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:11:29.831 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:11:29.832 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:11:29.832 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:11:29.832 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:11:29.832 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 98 M 2024-03-28T16:11:29.832 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:11:29.832 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:11:29.832 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:11:29.832 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:11:29.832 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:11:29.832 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 98 M 2024-03-28T16:11:29.832 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 450 M 2024-03-28T16:11:29.832 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:11:31.401 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-mon-debuginfo-18.2.2-1053.ge58b9979.el9.x8 15 MB/s | 98 MB 00:06 2024-03-28T16:11:31.402 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:11:31.402 INFO:teuthology.orchestra.run.smithi083.stdout:Total 15 MB/s | 98 MB 00:06 2024-03-28T16:11:31.403 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:11:31.407 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:11:31.407 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:11:32.020 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:11:32.020 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:11:35.113 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-mon-debuginfo-18.2.2-1053.ge58b9979.el9.x8 19 MB/s | 98 MB 00:05 2024-03-28T16:11:35.114 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:11:35.114 INFO:teuthology.orchestra.run.smithi007.stdout:Total 19 MB/s | 98 MB 00:05 2024-03-28T16:11:35.114 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:11:35.119 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:11:35.119 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:11:35.757 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:11:35.757 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:11:42.307 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:11:42.406 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mon-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:11:45.480 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:11:45.745 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mon-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:11:45.746 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:11:45.746 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:11:45.746 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:11:45.746 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:11:45.746 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:11:45.917 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-osd-debuginfo 2024-03-28T16:11:46.474 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:07:06 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:11:46.619 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:11:46.620 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:11:46.620 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:11:46.620 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:11:46.621 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:11:46.621 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 286 M 2024-03-28T16:11:46.621 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:11:46.621 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:11:46.621 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:11:46.621 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:11:46.621 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:11:46.621 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 286 M 2024-03-28T16:11:46.621 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 1.2 G 2024-03-28T16:11:46.621 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:11:46.634 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:11:46.737 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mon-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:11:49.730 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:11:50.010 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mon-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:11:50.010 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:11:50.010 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:11:50.010 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:11:50.010 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:11:50.010 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:11:50.338 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-osd-debuginfo 2024-03-28T16:11:50.912 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:59 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:11:51.066 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:11:51.067 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:11:51.067 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:11:51.067 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:11:51.067 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:11:51.067 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 286 M 2024-03-28T16:11:51.067 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:11:51.067 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:11:51.067 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:11:51.067 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:11:51.067 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:11:51.067 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 286 M 2024-03-28T16:11:51.068 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.2 G 2024-03-28T16:11:51.068 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:12:01.744 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-osd-debuginfo-18.2.2-1053.ge58b9979.el9.x8 19 MB/s | 286 MB 00:15 2024-03-28T16:12:01.744 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:12:01.745 INFO:teuthology.orchestra.run.smithi083.stdout:Total 19 MB/s | 286 MB 00:15 2024-03-28T16:12:01.745 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:12:01.749 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:12:01.749 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:12:03.457 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:12:03.457 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:12:09.487 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-osd-debuginfo-18.2.2-1053.ge58b9979.el9.x8 16 MB/s | 286 MB 00:18 2024-03-28T16:12:09.488 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:12:09.488 INFO:teuthology.orchestra.run.smithi007.stdout:Total 16 MB/s | 286 MB 00:18 2024-03-28T16:12:09.489 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:12:09.493 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:12:09.493 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:12:11.279 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:12:11.279 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:12:32.761 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:12:32.857 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-osd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:12:40.591 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:12:40.861 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-osd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:12:40.861 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:12:40.861 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:12:40.861 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:12:40.861 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:12:40.861 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:12:41.088 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-fuse-debuginfo 2024-03-28T16:12:41.647 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:08:01 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:12:41.796 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:12:41.797 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:12:41.797 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:12:41.797 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:12:41.797 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:12:41.797 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 5.8 M 2024-03-28T16:12:41.797 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:12:41.797 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:12:41.797 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:12:41.797 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:12:41.798 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:12:41.798 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 5.8 M 2024-03-28T16:12:41.798 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 28 M 2024-03-28T16:12:41.798 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:12:42.334 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:12:42.434 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-osd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:12:43.577 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-fuse-debuginfo-18.2.2-1053.ge58b9979.el9.x 3.3 MB/s | 5.8 MB 00:01 2024-03-28T16:12:43.578 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:12:43.578 INFO:teuthology.orchestra.run.smithi083.stdout:Total 3.3 MB/s | 5.8 MB 00:01 2024-03-28T16:12:43.578 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:12:43.582 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:12:43.583 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:12:43.653 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:12:43.654 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:12:44.384 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:12:44.527 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-fuse-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 1/1 2024-03-28T16:12:45.055 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 1/1 2024-03-28T16:12:45.321 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-fuse-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 1/1 2024-03-28T16:12:45.321 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:12:45.322 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:12:45.322 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:12:45.322 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:12:45.322 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:12:45.502 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados-devel-debuginfo 2024-03-28T16:12:46.062 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:08:06 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:12:46.211 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:12:46.212 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:12:46.212 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:12:46.212 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:12:46.212 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:12:46.213 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 108 k 2024-03-28T16:12:46.213 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:12:46.213 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:12:46.213 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:12:46.213 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:12:46.213 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:12:46.213 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 108 k 2024-03-28T16:12:46.213 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 440 k 2024-03-28T16:12:46.213 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:12:46.738 INFO:teuthology.orchestra.run.smithi083.stdout:librados-devel-debuginfo-18.2.2-1053.ge58b9979. 207 kB/s | 108 kB 00:00 2024-03-28T16:12:46.739 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:12:46.739 INFO:teuthology.orchestra.run.smithi083.stdout:Total 206 kB/s | 108 kB 00:00 2024-03-28T16:12:46.739 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:12:46.742 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:12:46.742 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:12:46.778 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:12:46.778 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:12:46.897 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:12:47.003 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librados-devel-debuginfo-2:18.2.2-1053.ge58b9979.el9 1/1 2024-03-28T16:12:47.408 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados-devel-debuginfo-2:18.2.2-1053.ge58b9979.el9 1/1 2024-03-28T16:12:47.667 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados-devel-debuginfo-2:18.2.2-1053.ge58b9979.el9 1/1 2024-03-28T16:12:47.668 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:12:47.668 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:12:47.668 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:12:47.668 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:12:47.668 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:12:47.878 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install libcephfs2-debuginfo 2024-03-28T16:12:48.436 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:08:08 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:12:48.585 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:12:48.586 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:12:48.586 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:12:48.586 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:12:48.586 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:12:48.586 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 5.1 M 2024-03-28T16:12:48.586 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:12:48.586 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:12:48.586 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:12:48.587 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:12:48.587 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:12:48.588 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 5.1 M 2024-03-28T16:12:48.588 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 24 M 2024-03-28T16:12:48.588 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:12:50.226 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:12:50.506 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-osd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:12:50.506 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:12:50.506 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:12:50.506 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:12:50.506 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:12:50.506 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:12:50.725 INFO:teuthology.orchestra.run.smithi083.stdout:libcephfs2-debuginfo-18.2.2-1053.ge58b9979.el9. 2.4 MB/s | 5.1 MB 00:02 2024-03-28T16:12:50.725 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:12:50.726 INFO:teuthology.orchestra.run.smithi083.stdout:Total 2.4 MB/s | 5.1 MB 00:02 2024-03-28T16:12:50.726 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:12:50.730 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-fuse-debuginfo 2024-03-28T16:12:50.731 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:12:50.731 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:12:50.796 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:12:50.797 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:12:51.302 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:00 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:12:51.450 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:12:51.457 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:12:51.458 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:12:51.458 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:12:51.458 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:12:51.458 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:12:51.458 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 5.8 M 2024-03-28T16:12:51.458 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:12:51.458 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:12:51.459 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:12:51.459 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:12:51.459 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:12:51.459 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 5.8 M 2024-03-28T16:12:51.459 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 28 M 2024-03-28T16:12:51.459 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:12:51.580 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs2-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86 1/1 2024-03-28T16:12:52.107 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86 1/1 2024-03-28T16:12:52.377 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs2-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86 1/1 2024-03-28T16:12:52.377 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:12:52.377 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:12:52.377 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:12:52.377 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:12:52.378 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:12:52.555 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados2-debuginfo 2024-03-28T16:12:52.919 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-fuse-debuginfo-18.2.2-1053.ge58b9979.el9.x 4.0 MB/s | 5.8 MB 00:01 2024-03-28T16:12:52.920 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:12:52.920 INFO:teuthology.orchestra.run.smithi007.stdout:Total 4.0 MB/s | 5.8 MB 00:01 2024-03-28T16:12:52.921 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:12:52.925 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:12:52.925 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:12:52.999 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:12:52.999 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:12:53.116 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:08:13 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:12:53.265 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:12:53.266 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:12:53.266 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:12:53.266 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:12:53.266 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:12:53.266 INFO:teuthology.orchestra.run.smithi083.stdout: librados2-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 45 M 2024-03-28T16:12:53.266 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:12:53.266 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:12:53.266 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:12:53.266 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:12:53.267 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:12:53.267 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 45 M 2024-03-28T16:12:53.267 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 256 M 2024-03-28T16:12:53.267 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:12:53.762 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:12:53.862 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-fuse-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 1/1 2024-03-28T16:12:54.409 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 1/1 2024-03-28T16:12:54.687 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-fuse-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 1/1 2024-03-28T16:12:54.687 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:12:54.687 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:12:54.687 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:12:54.687 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:12:54.687 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:12:54.887 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados-devel-debuginfo 2024-03-28T16:12:55.457 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:04 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:12:55.613 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:12:55.614 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:12:55.614 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:12:55.614 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:12:55.614 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:12:55.614 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 108 k 2024-03-28T16:12:55.614 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:12:55.614 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:12:55.614 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:12:55.614 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:12:55.614 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:12:55.614 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 108 k 2024-03-28T16:12:55.615 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 440 k 2024-03-28T16:12:55.615 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:12:56.139 INFO:teuthology.orchestra.run.smithi007.stdout:librados-devel-debuginfo-18.2.2-1053.ge58b9979. 207 kB/s | 108 kB 00:00 2024-03-28T16:12:56.139 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:12:56.139 INFO:teuthology.orchestra.run.smithi007.stdout:Total 207 kB/s | 108 kB 00:00 2024-03-28T16:12:56.140 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:12:56.143 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:12:56.143 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:12:56.179 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:12:56.179 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:12:56.300 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:12:56.415 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados-devel-debuginfo-2:18.2.2-1053.ge58b9979.el9 1/1 2024-03-28T16:12:56.893 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados-devel-debuginfo-2:18.2.2-1053.ge58b9979.el9 1/1 2024-03-28T16:12:57.161 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados-devel-debuginfo-2:18.2.2-1053.ge58b9979.el9 1/1 2024-03-28T16:12:57.161 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:12:57.161 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:12:57.161 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:12:57.161 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:12:57.161 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:12:57.488 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs2-debuginfo 2024-03-28T16:12:58.052 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:07 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:12:58.206 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:12:58.207 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:12:58.207 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:12:58.207 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:12:58.207 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:12:58.207 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 5.1 M 2024-03-28T16:12:58.207 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:12:58.207 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:12:58.207 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:12:58.207 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:12:58.207 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:12:58.208 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 5.1 M 2024-03-28T16:12:58.208 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 24 M 2024-03-28T16:12:58.208 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:12:59.053 INFO:teuthology.orchestra.run.smithi083.stdout:librados2-debuginfo-18.2.2-1053.ge58b9979.el9.x 7.8 MB/s | 45 MB 00:05 2024-03-28T16:12:59.053 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:12:59.054 INFO:teuthology.orchestra.run.smithi083.stdout:Total 7.8 MB/s | 45 MB 00:05 2024-03-28T16:12:59.054 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:12:59.058 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:12:59.058 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:12:59.362 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:12:59.362 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:12:59.775 INFO:teuthology.orchestra.run.smithi007.stdout:libcephfs2-debuginfo-18.2.2-1053.ge58b9979.el9. 3.3 MB/s | 5.1 MB 00:01 2024-03-28T16:12:59.776 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:12:59.777 INFO:teuthology.orchestra.run.smithi007.stdout:Total 3.3 MB/s | 5.1 MB 00:01 2024-03-28T16:12:59.777 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:12:59.781 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:12:59.781 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:12:59.850 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:12:59.851 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:13:00.520 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:13:00.627 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs2-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86 1/1 2024-03-28T16:13:01.130 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86 1/1 2024-03-28T16:13:01.415 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs2-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86 1/1 2024-03-28T16:13:01.416 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:01.416 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:13:01.416 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:13:01.416 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:01.416 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:13:01.626 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados2-debuginfo 2024-03-28T16:13:02.196 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:11 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:13:02.351 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:13:02.352 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:02.352 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:13:02.352 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:02.352 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:13:02.352 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 45 M 2024-03-28T16:13:02.352 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:02.353 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:13:02.353 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:02.353 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:13:02.353 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:02.353 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 45 M 2024-03-28T16:13:02.353 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 256 M 2024-03-28T16:13:02.353 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:13:04.501 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:13:04.593 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librados2-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 1/1 2024-03-28T16:13:05.303 INFO:teuthology.orchestra.run.smithi007.stdout:librados2-debuginfo-18.2.2-1053.ge58b9979.el9.x 15 MB/s | 45 MB 00:02 2024-03-28T16:13:05.304 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:13:05.304 INFO:teuthology.orchestra.run.smithi007.stdout:Total 15 MB/s | 45 MB 00:02 2024-03-28T16:13:05.304 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:13:05.308 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:13:05.309 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:13:05.628 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:13:05.628 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:13:06.529 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 1/1 2024-03-28T16:13:06.788 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 1/1 2024-03-28T16:13:06.789 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:06.789 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:13:06.789 INFO:teuthology.orchestra.run.smithi083.stdout: librados2-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:13:06.789 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:06.789 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:13:06.993 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librbd1-debuginfo 2024-03-28T16:13:07.570 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:08:27 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:13:07.730 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:13:07.731 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:07.731 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:13:07.731 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:07.731 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:13:07.732 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 50 M 2024-03-28T16:13:07.732 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:07.732 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:13:07.732 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:07.732 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:13:07.732 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:07.732 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 50 M 2024-03-28T16:13:07.732 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 291 M 2024-03-28T16:13:07.732 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:13:10.413 INFO:teuthology.orchestra.run.smithi083.stdout:librbd1-debuginfo-18.2.2-1053.ge58b9979.el9.x86 19 MB/s | 50 MB 00:02 2024-03-28T16:13:10.414 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:13:10.415 INFO:teuthology.orchestra.run.smithi083.stdout:Total 19 MB/s | 50 MB 00:02 2024-03-28T16:13:10.415 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:13:10.419 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:13:10.419 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:13:10.756 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:13:10.756 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:13:11.079 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:13:11.179 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados2-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 1/1 2024-03-28T16:13:13.204 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 1/1 2024-03-28T16:13:13.477 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_ 1/1 2024-03-28T16:13:13.478 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:13.478 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:13:13.478 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:13:13.478 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:13.478 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:13:13.676 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librbd1-debuginfo 2024-03-28T16:13:14.247 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:23 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:13:14.401 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:13:14.402 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:14.402 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:13:14.402 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:14.402 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:13:14.402 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 50 M 2024-03-28T16:13:14.402 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:14.402 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:13:14.402 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:14.402 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:13:14.402 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:14.402 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 50 M 2024-03-28T16:13:14.402 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 291 M 2024-03-28T16:13:14.403 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:13:16.562 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:13:16.879 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librbd1-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:13:17.455 INFO:teuthology.orchestra.run.smithi007.stdout:librbd1-debuginfo-18.2.2-1053.ge58b9979.el9.x86 17 MB/s | 50 MB 00:03 2024-03-28T16:13:17.456 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:13:17.457 INFO:teuthology.orchestra.run.smithi007.stdout:Total 17 MB/s | 50 MB 00:03 2024-03-28T16:13:17.457 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:13:17.461 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:13:17.461 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:13:17.812 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:13:17.812 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:13:19.022 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:13:19.286 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:13:19.286 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:19.286 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:13:19.286 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:13:19.286 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:19.286 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:13:19.477 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-cephfs-debuginfo 2024-03-28T16:13:20.043 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:08:40 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:13:20.195 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:13:20.196 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:20.196 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:13:20.196 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:20.196 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:13:20.196 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 615 k 2024-03-28T16:13:20.196 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:20.196 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:13:20.196 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:20.197 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:13:20.197 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:20.197 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 615 k 2024-03-28T16:13:20.197 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 2.1 M 2024-03-28T16:13:20.197 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:13:20.475 INFO:teuthology.orchestra.run.smithi083.stdout:python3-cephfs-debuginfo-18.2.2-1053.ge58b9979. 2.2 MB/s | 615 kB 00:00 2024-03-28T16:13:20.476 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:13:20.476 INFO:teuthology.orchestra.run.smithi083.stdout:Total 2.1 MB/s | 615 kB 00:00 2024-03-28T16:13:20.476 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:13:20.479 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:13:20.479 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:13:20.517 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:13:20.517 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:13:20.684 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:13:20.799 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cephfs-debuginfo-2:18.2.2-1053.ge58b9979.el9 1/1 2024-03-28T16:13:21.146 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.2.2-1053.ge58b9979.el9 1/1 2024-03-28T16:13:21.410 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cephfs-debuginfo-2:18.2.2-1053.ge58b9979.el9 1/1 2024-03-28T16:13:21.411 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:21.411 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:13:21.411 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:13:21.411 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:21.411 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:13:21.596 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rados-debuginfo 2024-03-28T16:13:22.155 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:08:42 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:13:22.307 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:13:22.307 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:22.307 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:13:22.307 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:22.307 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:13:22.307 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 768 k 2024-03-28T16:13:22.307 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:22.308 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:13:22.308 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:22.308 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:13:22.308 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:22.308 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 768 k 2024-03-28T16:13:22.308 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 2.7 M 2024-03-28T16:13:22.308 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:13:22.778 INFO:teuthology.orchestra.run.smithi083.stdout:python3-rados-debuginfo-18.2.2-1053.ge58b9979.e 1.6 MB/s | 768 kB 00:00 2024-03-28T16:13:22.779 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:13:22.779 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.6 MB/s | 768 kB 00:00 2024-03-28T16:13:22.780 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:13:22.783 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:13:22.783 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:13:22.821 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:13:22.821 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:13:22.999 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:13:23.103 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rados-debuginfo-2:18.2.2-1053.ge58b9979.el9. 1/1 2024-03-28T16:13:23.489 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: python3-rados-debuginfo-2:18.2.2-1053.ge58b9979.el9. 1/1 2024-03-28T16:13:23.747 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rados-debuginfo-2:18.2.2-1053.ge58b9979.el9. 1/1 2024-03-28T16:13:23.747 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:23.748 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:13:23.748 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:13:23.748 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:23.748 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:13:23.931 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:13:23.947 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rbd-debuginfo 2024-03-28T16:13:24.031 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librbd1-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:13:24.497 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:08:44 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:13:24.645 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:13:24.646 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:24.646 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:13:24.647 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:24.647 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:13:24.647 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 1.1 M 2024-03-28T16:13:24.647 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:24.647 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:13:24.647 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:24.647 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:13:24.647 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:24.647 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 1.1 M 2024-03-28T16:13:24.647 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 4.3 M 2024-03-28T16:13:24.647 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:13:25.157 INFO:teuthology.orchestra.run.smithi083.stdout:python3-rbd-debuginfo-18.2.2-1053.ge58b9979.el9 2.2 MB/s | 1.1 MB 00:00 2024-03-28T16:13:25.157 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:13:25.158 INFO:teuthology.orchestra.run.smithi083.stdout:Total 2.2 MB/s | 1.1 MB 00:00 2024-03-28T16:13:25.158 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:13:25.161 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:13:25.161 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:13:25.202 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:13:25.202 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:13:25.419 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:13:25.530 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rbd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x8 1/1 2024-03-28T16:13:25.901 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x8 1/1 2024-03-28T16:13:26.132 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:13:26.160 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rbd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x8 1/1 2024-03-28T16:13:26.160 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:26.160 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:13:26.160 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:13:26.160 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:26.161 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:13:26.364 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rgw-debuginfo 2024-03-28T16:13:26.397 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:13:26.397 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:26.397 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:13:26.397 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:13:26.397 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:26.397 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:13:26.612 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-cephfs-debuginfo 2024-03-28T16:13:26.920 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:08:46 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:13:27.068 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:13:27.069 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:27.069 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:13:27.069 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:27.069 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:13:27.069 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 337 k 2024-03-28T16:13:27.070 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:27.070 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:13:27.070 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:27.070 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:13:27.070 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:27.070 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 337 k 2024-03-28T16:13:27.070 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 1.0 M 2024-03-28T16:13:27.070 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:13:27.182 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:36 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:13:27.319 INFO:teuthology.orchestra.run.smithi083.stdout:python3-rgw-debuginfo-18.2.2-1053.ge58b9979.el9 1.3 MB/s | 337 kB 00:00 2024-03-28T16:13:27.320 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:13:27.320 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.3 MB/s | 337 kB 00:00 2024-03-28T16:13:27.320 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:13:27.323 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:13:27.323 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:13:27.339 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:13:27.340 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:27.340 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:13:27.340 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:27.340 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:13:27.340 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 615 k 2024-03-28T16:13:27.341 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:27.341 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:13:27.341 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:27.341 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:13:27.341 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:27.341 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 615 k 2024-03-28T16:13:27.341 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 2.1 M 2024-03-28T16:13:27.341 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:13:27.358 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:13:27.359 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:13:27.498 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:13:27.608 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rgw-debuginfo-2:18.2.2-1053.ge58b9979.el9.x8 1/1 2024-03-28T16:13:27.627 INFO:teuthology.orchestra.run.smithi007.stdout:python3-cephfs-debuginfo-18.2.2-1053.ge58b9979. 2.1 MB/s | 615 kB 00:00 2024-03-28T16:13:27.628 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:13:27.628 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.1 MB/s | 615 kB 00:00 2024-03-28T16:13:27.628 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:13:27.629 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:13:27.629 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:13:27.668 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:13:27.668 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:13:27.837 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:13:27.951 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cephfs-debuginfo-2:18.2.2-1053.ge58b9979.el9 1/1 2024-03-28T16:13:27.963 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.2.2-1053.ge58b9979.el9.x8 1/1 2024-03-28T16:13:28.223 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rgw-debuginfo-2:18.2.2-1053.ge58b9979.el9.x8 1/1 2024-03-28T16:13:28.223 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:28.223 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:13:28.223 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:13:28.224 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:28.224 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:13:28.375 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.2.2-1053.ge58b9979.el9 1/1 2024-03-28T16:13:28.420 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-fuse-debuginfo 2024-03-28T16:13:28.664 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cephfs-debuginfo-2:18.2.2-1053.ge58b9979.el9 1/1 2024-03-28T16:13:28.665 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:28.665 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:13:28.665 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:13:28.665 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:28.665 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:13:28.864 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rados-debuginfo 2024-03-28T16:13:28.977 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:08:48 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:13:29.125 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:13:29.126 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:29.126 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:13:29.126 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:29.126 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:13:29.126 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 511 k 2024-03-28T16:13:29.126 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:29.126 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:13:29.126 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:29.127 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:13:29.127 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:29.127 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 511 k 2024-03-28T16:13:29.127 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 2.5 M 2024-03-28T16:13:29.127 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:13:29.422 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-fuse-debuginfo-18.2.2-1053.ge58b9979.el9.x8 1.7 MB/s | 511 kB 00:00 2024-03-28T16:13:29.422 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:13:29.422 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.7 MB/s | 511 kB 00:00 2024-03-28T16:13:29.423 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:13:29.426 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:13:29.426 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:13:29.434 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:38 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:13:29.462 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:13:29.462 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:13:29.592 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:13:29.592 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:29.592 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:13:29.593 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:29.593 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:13:29.593 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 768 k 2024-03-28T16:13:29.593 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:29.593 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:13:29.593 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:29.593 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:13:29.593 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:29.594 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 768 k 2024-03-28T16:13:29.594 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 2.7 M 2024-03-28T16:13:29.594 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:13:29.621 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:13:29.722 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-fuse-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:13:29.979 INFO:teuthology.orchestra.run.smithi007.stdout:python3-rados-debuginfo-18.2.2-1053.ge58b9979.e 1.9 MB/s | 768 kB 00:00 2024-03-28T16:13:29.979 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:13:29.979 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.9 MB/s | 768 kB 00:00 2024-03-28T16:13:29.980 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:13:29.983 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:13:29.983 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:13:30.023 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:13:30.023 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:13:30.109 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:13:30.212 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:13:30.313 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rados-debuginfo-2:18.2.2-1053.ge58b9979.el9. 1/1 2024-03-28T16:13:30.371 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-fuse-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:13:30.371 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:30.371 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:13:30.371 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:13:30.371 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:30.371 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:13:30.556 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-mirror-debuginfo 2024-03-28T16:13:30.832 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: python3-rados-debuginfo-2:18.2.2-1053.ge58b9979.el9. 1/1 2024-03-28T16:13:31.105 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rados-debuginfo-2:18.2.2-1053.ge58b9979.el9. 1/1 2024-03-28T16:13:31.106 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:31.106 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:13:31.106 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:13:31.106 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:31.106 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:13:31.109 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:08:51 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:13:31.257 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:13:31.258 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:31.258 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:13:31.258 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:31.258 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:13:31.258 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 50 M 2024-03-28T16:13:31.258 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:31.258 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:13:31.258 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:31.258 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:13:31.258 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:31.258 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 50 M 2024-03-28T16:13:31.258 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 281 M 2024-03-28T16:13:31.259 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:13:31.317 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rbd-debuginfo 2024-03-28T16:13:31.887 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:40 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:13:32.041 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:13:32.042 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:32.042 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:13:32.042 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:32.042 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:13:32.042 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 1.1 M 2024-03-28T16:13:32.042 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:32.042 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:13:32.042 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:32.042 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:13:32.042 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:32.042 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 1.1 M 2024-03-28T16:13:32.042 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 4.3 M 2024-03-28T16:13:32.043 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:13:33.190 INFO:teuthology.orchestra.run.smithi007.stdout:python3-rbd-debuginfo-18.2.2-1053.ge58b9979.el9 1.0 MB/s | 1.1 MB 00:01 2024-03-28T16:13:33.190 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:13:33.191 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.0 MB/s | 1.1 MB 00:01 2024-03-28T16:13:33.191 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:13:33.195 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:13:33.195 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:13:33.241 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:13:33.241 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:13:33.472 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:13:33.583 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rbd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x8 1/1 2024-03-28T16:13:33.982 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x8 1/1 2024-03-28T16:13:33.983 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-mirror-debuginfo-18.2.2-1053.ge58b9979.el9. 18 MB/s | 50 MB 00:02 2024-03-28T16:13:33.984 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:13:33.984 INFO:teuthology.orchestra.run.smithi083.stdout:Total 18 MB/s | 50 MB 00:02 2024-03-28T16:13:33.984 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:13:33.988 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:13:33.988 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:13:34.271 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rbd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x8 1/1 2024-03-28T16:13:34.271 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:34.271 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:13:34.271 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:13:34.271 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:34.271 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:13:34.316 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:13:34.316 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:13:34.490 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rgw-debuginfo 2024-03-28T16:13:35.061 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:44 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:13:35.215 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:13:35.216 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:35.216 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:13:35.216 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:35.216 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:13:35.216 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 337 k 2024-03-28T16:13:35.216 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:35.216 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:13:35.216 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:35.216 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:13:35.216 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:35.216 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 337 k 2024-03-28T16:13:35.216 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.0 M 2024-03-28T16:13:35.216 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:13:35.468 INFO:teuthology.orchestra.run.smithi007.stdout:python3-rgw-debuginfo-18.2.2-1053.ge58b9979.el9 1.3 MB/s | 337 kB 00:00 2024-03-28T16:13:35.469 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:13:35.469 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.3 MB/s | 337 kB 00:00 2024-03-28T16:13:35.469 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:13:35.473 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:13:35.473 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:13:35.509 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:13:35.510 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:13:35.650 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:13:35.752 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rgw-debuginfo-2:18.2.2-1053.ge58b9979.el9.x8 1/1 2024-03-28T16:13:36.097 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.2.2-1053.ge58b9979.el9.x8 1/1 2024-03-28T16:13:36.371 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rgw-debuginfo-2:18.2.2-1053.ge58b9979.el9.x8 1/1 2024-03-28T16:13:36.371 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:36.371 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:13:36.372 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:13:36.372 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:36.372 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:13:36.578 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-fuse-debuginfo 2024-03-28T16:13:37.149 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:46 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:13:37.304 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:13:37.305 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:37.305 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:13:37.305 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:37.305 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:13:37.305 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 511 k 2024-03-28T16:13:37.305 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:37.305 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:13:37.305 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:37.305 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:13:37.305 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:37.305 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 511 k 2024-03-28T16:13:37.305 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 2.5 M 2024-03-28T16:13:37.306 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:13:37.572 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-fuse-debuginfo-18.2.2-1053.ge58b9979.el9.x8 1.9 MB/s | 511 kB 00:00 2024-03-28T16:13:37.572 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:13:37.572 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.9 MB/s | 511 kB 00:00 2024-03-28T16:13:37.572 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:13:37.576 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:13:37.576 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:13:37.615 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:13:37.615 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:13:37.782 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:13:37.891 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-fuse-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:13:38.305 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:13:38.570 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-fuse-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_6 1/1 2024-03-28T16:13:38.570 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:38.570 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:13:38.570 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:13:38.571 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:38.571 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:13:38.761 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-mirror-debuginfo 2024-03-28T16:13:39.329 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:08:48 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:13:39.484 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:13:39.486 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:39.486 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:13:39.486 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:39.486 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:13:39.487 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 50 M 2024-03-28T16:13:39.487 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:39.487 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:13:39.488 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:39.488 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:13:39.488 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:39.488 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 50 M 2024-03-28T16:13:39.489 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 281 M 2024-03-28T16:13:39.489 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:13:40.006 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:13:40.140 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-mirror-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86 1/1 2024-03-28T16:13:41.908 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-mirror-debuginfo-18.2.2-1053.ge58b9979.el9. 21 MB/s | 50 MB 00:02 2024-03-28T16:13:41.909 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:13:41.909 INFO:teuthology.orchestra.run.smithi007.stdout:Total 21 MB/s | 50 MB 00:02 2024-03-28T16:13:41.909 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:13:41.914 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:13:41.914 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:13:42.117 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86 1/1 2024-03-28T16:13:42.258 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:13:42.258 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:13:42.388 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-mirror-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86 1/1 2024-03-28T16:13:42.388 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:42.388 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:13:42.388 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:13:42.388 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:42.388 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:13:42.587 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-nbd-debuginfo 2024-03-28T16:13:43.145 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:09:03 ago on Thu 28 Mar 2024 04:04:40 PM UTC. 2024-03-28T16:13:43.294 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-28T16:13:43.294 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:43.294 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-28T16:13:43.294 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:43.295 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-28T16:13:43.295 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 1.3 M 2024-03-28T16:13:43.295 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:43.295 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-28T16:13:43.295 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-28T16:13:43.295 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-28T16:13:43.295 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:43.295 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 1.3 M 2024-03-28T16:13:43.295 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 5.5 M 2024-03-28T16:13:43.295 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-28T16:13:43.635 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-nbd-debuginfo-18.2.2-1053.ge58b9979.el9.x86 3.7 MB/s | 1.3 MB 00:00 2024-03-28T16:13:43.636 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:13:43.636 INFO:teuthology.orchestra.run.smithi083.stdout:Total 3.7 MB/s | 1.3 MB 00:00 2024-03-28T16:13:43.636 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-28T16:13:43.640 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-28T16:13:43.640 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-28T16:13:43.680 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-28T16:13:43.680 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-28T16:13:43.911 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-28T16:13:44.017 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-nbd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:13:44.412 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:13:44.679 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-nbd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:13:44.679 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:44.679 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-28T16:13:44.679 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:13:44.679 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:44.679 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-28T16:13:44.871 DEBUG:teuthology.parallel:result is None 2024-03-28T16:13:48.289 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:13:48.394 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-mirror-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86 1/1 2024-03-28T16:13:50.363 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86 1/1 2024-03-28T16:13:50.627 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-mirror-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86 1/1 2024-03-28T16:13:50.627 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:50.627 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:13:50.627 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:13:50.627 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:50.627 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:13:50.849 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-nbd-debuginfo 2024-03-28T16:13:51.421 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:09:00 ago on Thu 28 Mar 2024 04:04:51 PM UTC. 2024-03-28T16:13:51.577 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-28T16:13:51.578 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:51.578 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-28T16:13:51.578 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:51.578 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-28T16:13:51.578 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-debuginfo x86_64 2:18.2.2-1053.ge58b9979.el9 ceph 1.3 M 2024-03-28T16:13:51.578 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:51.578 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-28T16:13:51.578 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-28T16:13:51.578 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-28T16:13:51.578 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:51.578 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 1.3 M 2024-03-28T16:13:51.578 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 5.5 M 2024-03-28T16:13:51.579 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-28T16:13:52.010 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-nbd-debuginfo-18.2.2-1053.ge58b9979.el9.x86 2.9 MB/s | 1.3 MB 00:00 2024-03-28T16:13:52.011 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-28T16:13:52.011 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.9 MB/s | 1.3 MB 00:00 2024-03-28T16:13:52.011 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-28T16:13:52.014 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-28T16:13:52.014 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-28T16:13:52.057 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-28T16:13:52.057 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-28T16:13:52.305 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-28T16:13:52.412 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-nbd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:13:52.827 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:13:53.100 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-nbd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 1/1 2024-03-28T16:13:53.100 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:53.101 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-28T16:13:53.101 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-debuginfo-2:18.2.2-1053.ge58b9979.el9.x86_64 2024-03-28T16:13:53.101 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:53.101 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-28T16:13:53.296 DEBUG:teuthology.parallel:result is None 2024-03-28T16:13:53.297 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e58b9979847f029f7ab1514b5f5f99b9f03d97f6 2024-03-28T16:13:53.441 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T16:13:53.461 INFO:teuthology.orchestra.run.smithi007.stdout:18.2.2-1053.ge58b9979.el9 2024-03-28T16:13:53.462 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1053.ge58b9979.el9 2024-03-28T16:13:53.462 INFO:teuthology.task.install:The correct ceph version 18.2.2-1053.ge58b9979 is installed. 2024-03-28T16:13:53.464 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e58b9979847f029f7ab1514b5f5f99b9f03d97f6 2024-03-28T16:13:53.599 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T16:13:53.623 INFO:teuthology.orchestra.run.smithi083.stdout:18.2.2-1053.ge58b9979.el9 2024-03-28T16:13:53.624 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1053.ge58b9979.el9 2024-03-28T16:13:53.624 INFO:teuthology.task.install:The correct ceph version 18.2.2-1053.ge58b9979 is installed. 2024-03-28T16:13:53.626 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-28T16:13:53.626 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:13:53.626 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T16:13:53.653 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-28T16:13:53.653 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T16:13:53.694 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-28T16:13:53.694 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:13:53.695 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T16:13:53.721 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T16:13:53.788 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-28T16:13:53.788 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T16:13:53.820 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T16:13:53.888 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-28T16:13:53.888 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:13:53.888 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T16:13:53.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T16:13:53.981 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-28T16:13:53.981 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T16:13:54.011 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T16:13:54.079 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-28T16:13:54.079 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:13:54.079 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T16:13:54.105 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T16:13:54.172 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-28T16:13:54.172 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T16:13:54.201 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T16:13:54.268 INFO:teuthology.run_tasks:Running task ceph... 2024-03-28T16:13:54.337 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-28T16:13:54.337 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /var/log/ceph 2024-03-28T16:13:54.339 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 777 /var/log/ceph 2024-03-28T16:13:54.366 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-28T16:13:54.366 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-28T16:13:54.406 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-28T16:13:54.435 INFO:tasks.ceph:Creating extra log directories... 2024-03-28T16:13:54.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-28T16:13:54.473 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-28T16:13:54.536 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-28T16:13:54.536 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-28T16:13:54.537 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-03-28_15:43:50-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7628301', 'branch': 'wip-yuri4-testing-2024-03-26-1132-reef', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7628301', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-03-28_15:43:50-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2024-03-26-1132-reef'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'e58b9979847f029f7ab1514b5f5f99b9f03d97f6', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'e58b9979847f029f7ab1514b5f5f99b9f03d97f6'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2024-03-26-1132-reef', 'sha1': 'e58b9979847f029f7ab1514b5f5f99b9f03d97f6'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6904, 'sha1': 'e58b9979847f029f7ab1514b5f5f99b9f03d97f6', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri4-testing-2024-03-26-1132-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'e58b9979847f029f7ab1514b5f5f99b9f03d97f6', 'targets': {'smithi007.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPSlIhOqcL4HPPLuorlvP91WM/eGDMhSxwDVQxUeX+rJZctwWzPZuqCmEEZMlGLS1+H4aGa0KHzKTPaqWlGLt5A=', 'smithi083.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPEEzb0wEA3mhYdRzG0yLCtBdbNckMpzYgietdNUhBv07DwgXUIwEb937mdiXay9pYyp/gyhuKgZaVKV1FcQxNA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-28_15:43:50', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-28T16:13:54.537 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-28T16:13:54.540 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-28T16:13:54.556 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/run/ceph 2024-03-28T16:13:54.594 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/run/ceph 2024-03-28T16:13:54.622 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:13:54.622 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T16:13:54.674 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T16:13:54.674 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2024-03-28T16:13:54.728 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T16:13:54.728 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T16:13:54.728 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-28T16:13:54.728 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T16:13:54.728 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T16:13:54.728 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-28 16:06:38.682764891 +0000 2024-03-28T16:13:54.729 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-28 16:03:46.420444160 +0000 2024-03-28T16:13:54.729 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-28 16:03:46.420444160 +0000 2024-03-28T16:13:54.729 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-28 16:03:46.420444160 +0000 2024-03-28T16:13:54.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T16:13:54.794 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-28T16:13:54.794 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-28T16:13:54.794 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 8.0044e-05 s, 6.4 MB/s 2024-03-28T16:13:54.795 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T16:13:54.851 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2024-03-28T16:13:54.905 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T16:13:54.905 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T16:13:54.905 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-28T16:13:54.905 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T16:13:54.905 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T16:13:54.905 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-28 16:06:38.683764875 +0000 2024-03-28T16:13:54.905 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-28 16:03:46.754438801 +0000 2024-03-28T16:13:54.906 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-28 16:03:46.754438801 +0000 2024-03-28T16:13:54.906 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-28 16:03:46.754438801 +0000 2024-03-28T16:13:54.906 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T16:13:54.970 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-28T16:13:54.970 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-28T16:13:54.970 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 9.4334e-05 s, 5.4 MB/s 2024-03-28T16:13:54.972 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T16:13:55.027 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2024-03-28T16:13:55.081 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T16:13:55.081 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T16:13:55.081 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-28T16:13:55.081 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T16:13:55.081 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T16:13:55.081 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-28 16:06:38.683764875 +0000 2024-03-28T16:13:55.081 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-28 16:03:47.085433490 +0000 2024-03-28T16:13:55.082 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-28 16:03:47.085433490 +0000 2024-03-28T16:13:55.082 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-28 16:03:47.085433490 +0000 2024-03-28T16:13:55.082 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T16:13:55.145 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-28T16:13:55.145 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-28T16:13:55.146 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 8.9507e-05 s, 5.7 MB/s 2024-03-28T16:13:55.147 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T16:13:55.202 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2024-03-28T16:13:55.256 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T16:13:55.256 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T16:13:55.256 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-28T16:13:55.256 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T16:13:55.256 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T16:13:55.256 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-28 16:06:38.683764875 +0000 2024-03-28T16:13:55.256 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-28 16:03:47.415428195 +0000 2024-03-28T16:13:55.256 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-28 16:03:47.415428195 +0000 2024-03-28T16:13:55.256 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-28 16:03:47.415428195 +0000 2024-03-28T16:13:55.257 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T16:13:55.320 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-28T16:13:55.320 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-28T16:13:55.320 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 9.2719e-05 s, 5.5 MB/s 2024-03-28T16:13:55.321 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T16:13:55.378 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T16:13:55.378 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-28T16:13:55.378 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T16:13:55.395 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T16:13:55.395 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_1 2024-03-28T16:13:55.451 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T16:13:55.451 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T16:13:55.451 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-28T16:13:55.452 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T16:13:55.452 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T16:13:55.452 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-28 16:06:26.470956738 +0000 2024-03-28T16:13:55.452 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-28 16:01:34.826553247 +0000 2024-03-28T16:13:55.452 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-28 16:01:34.826553247 +0000 2024-03-28T16:13:55.452 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-28 16:01:34.826553247 +0000 2024-03-28T16:13:55.452 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T16:13:55.518 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-28T16:13:55.518 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-28T16:13:55.519 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000252531 s, 2.0 MB/s 2024-03-28T16:13:55.520 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T16:13:55.577 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_2 2024-03-28T16:13:55.634 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T16:13:55.634 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T16:13:55.634 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-28T16:13:55.634 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T16:13:55.634 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T16:13:55.635 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-28 16:06:26.470956738 +0000 2024-03-28T16:13:55.635 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-28 16:01:35.162545068 +0000 2024-03-28T16:13:55.635 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-28 16:01:35.162545068 +0000 2024-03-28T16:13:55.635 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-28 16:01:35.162545068 +0000 2024-03-28T16:13:55.635 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T16:13:55.701 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-28T16:13:55.701 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-28T16:13:55.701 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000235679 s, 2.2 MB/s 2024-03-28T16:13:55.702 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T16:13:55.758 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_3 2024-03-28T16:13:55.812 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T16:13:55.813 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T16:13:55.813 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-28T16:13:55.813 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T16:13:55.813 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T16:13:55.813 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-28 16:06:26.470956738 +0000 2024-03-28T16:13:55.813 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-28 16:01:35.505536719 +0000 2024-03-28T16:13:55.813 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-28 16:01:35.505536719 +0000 2024-03-28T16:13:55.813 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-28 16:01:35.505536719 +0000 2024-03-28T16:13:55.813 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T16:13:55.878 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-28T16:13:55.878 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-28T16:13:55.878 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000221924 s, 2.3 MB/s 2024-03-28T16:13:55.879 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T16:13:55.935 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_4 2024-03-28T16:13:55.990 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T16:13:55.990 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T16:13:55.990 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-28T16:13:55.990 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T16:13:55.990 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T16:13:55.990 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-28 16:06:26.471956714 +0000 2024-03-28T16:13:55.990 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-28 16:01:35.840528565 +0000 2024-03-28T16:13:55.990 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-28 16:01:35.840528565 +0000 2024-03-28T16:13:55.990 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-28 16:01:35.840528565 +0000 2024-03-28T16:13:55.991 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T16:13:56.054 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-28T16:13:56.055 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-28T16:13:56.055 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000208243 s, 2.5 MB/s 2024-03-28T16:13:56.056 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T16:13:56.111 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T16:13:56.111 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi083.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-28T16:13:56.111 INFO:tasks.ceph:Generating config... 2024-03-28T16:13:56.112 INFO:tasks.ceph:[client] debug monc = 20 2024-03-28T16:13:56.113 INFO:tasks.ceph:[client] debug ms = 1 2024-03-28T16:13:56.113 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-28T16:13:56.113 INFO:tasks.ceph:[client] debug rados = 20 2024-03-28T16:13:56.113 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-28T16:13:56.113 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-28T16:13:56.113 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-28T16:13:56.113 INFO:tasks.ceph:[global] ms type = async 2024-03-28T16:13:56.113 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-28T16:13:56.113 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-28T16:13:56.113 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-28T16:13:56.113 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-28T16:13:56.113 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-28T16:13:56.114 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-28T16:13:56.114 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-28T16:13:56.114 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-28T16:13:56.114 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-28T16:13:56.114 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-28T16:13:56.114 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-28T16:13:56.114 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-28T16:13:56.114 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-28T16:13:56.114 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-28T16:13:56.114 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-28T16:13:56.114 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-28T16:13:56.114 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-28T16:13:56.115 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-28T16:13:56.115 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-28T16:13:56.115 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-28T16:13:56.115 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-28T16:13:56.115 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-28T16:13:56.115 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-28T16:13:56.115 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-28T16:13:56.115 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-28T16:13:56.115 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-28T16:13:56.115 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-28T16:13:56.115 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-28T16:13:56.115 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-28T16:13:56.115 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-28T16:13:56.115 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-28T16:13:56.116 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-28T16:13:56.116 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-28T16:13:56.116 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-28T16:13:56.116 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-28T16:13:56.116 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-28T16:13:56.116 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-28T16:13:56.116 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-28T16:13:56.116 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-28T16:13:56.116 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-28T16:13:56.116 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-28T16:13:56.116 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-28T16:13:56.116 INFO:tasks.ceph:Setting up mon.a... 2024-03-28T16:13:56.117 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-28T16:13:56.166 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/ceph.keyring 2024-03-28T16:13:56.169 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-28T16:13:56.214 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-28T16:13:56.281 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.7'), ('mon.c', '[v2:172.21.15.7:3301,v1:172.21.15.7:6790]'), ('mon.b', '172.21.15.83')] 2024-03-28T16:13:56.281 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.7,[v2:172.21.15.7:3301,v1:172.21.15.7:6790],172.21.15.83', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-28T16:13:56.282 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:13:56.282 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-28T16:13:56.335 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.7 --addv c '[v2:172.21.15.7:3301,v1:172.21.15.7:6790]' --add b 172.21.15.83 --print /home/ubuntu/cephtest/ceph.monmap 2024-03-28T16:13:56.420 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-28T16:13:56.420 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: generated fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:13:56.420 INFO:teuthology.orchestra.run.smithi007.stdout:setting min_mon_release = pacific 2024-03-28T16:13:56.420 INFO:teuthology.orchestra.run.smithi007.stdout:epoch 0 2024-03-28T16:13:56.420 INFO:teuthology.orchestra.run.smithi007.stdout:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:13:56.420 INFO:teuthology.orchestra.run.smithi007.stdout:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:13:56.420 INFO:teuthology.orchestra.run.smithi007.stdout:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:13:56.420 INFO:teuthology.orchestra.run.smithi007.stdout:min_mon_release 16 (pacific) 2024-03-28T16:13:56.420 INFO:teuthology.orchestra.run.smithi007.stdout:election_strategy: 1 2024-03-28T16:13:56.420 INFO:teuthology.orchestra.run.smithi007.stdout:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:13:56.421 INFO:teuthology.orchestra.run.smithi007.stdout:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:13:56.421 INFO:teuthology.orchestra.run.smithi007.stdout:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:13:56.421 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-28T16:13:56.422 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-28T16:13:56.436 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 808ff1a1-2b17-4c35-a89a-340dbbf674ee... 2024-03-28T16:13:56.437 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-28T16:13:56.479 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-28T16:13:56.526 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2024-03-28T16:13:56.526 INFO:teuthology.orchestra.run.smithi007.stdout: chdir = "" 2024-03-28T16:13:56.526 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-28T16:13:56.526 INFO:teuthology.orchestra.run.smithi007.stdout: auth supported = cephx 2024-03-28T16:13:56.526 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.526 INFO:teuthology.orchestra.run.smithi007.stdout: filestore xattr use omap = true 2024-03-28T16:13:56.527 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.527 INFO:teuthology.orchestra.run.smithi007.stdout: mon clock drift allowed = 1.000 2024-03-28T16:13:56.527 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.527 INFO:teuthology.orchestra.run.smithi007.stdout: osd crush chooseleaf type = 0 2024-03-28T16:13:56.527 INFO:teuthology.orchestra.run.smithi007.stdout: auth debug = true 2024-03-28T16:13:56.527 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.527 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on old message = true 2024-03-28T16:13:56.527 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on bug = true 2024-03-28T16:13:56.527 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.527 INFO:teuthology.orchestra.run.smithi007.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-28T16:13:56.527 INFO:teuthology.orchestra.run.smithi007.stdout: mon pg warn max object skew = 0 2024-03-28T16:13:56.527 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.527 INFO:teuthology.orchestra.run.smithi007.stdout: # disable pg_autoscaler by default for new pools 2024-03-28T16:13:56.527 INFO:teuthology.orchestra.run.smithi007.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-28T16:13:56.527 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.528 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default size = 2 2024-03-28T16:13:56.528 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.528 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow primary affinity = true 2024-03-28T16:13:56.528 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow pg remap = true 2024-03-28T16:13:56.528 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on legacy crush tunables = false 2024-03-28T16:13:56.528 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on crush straw calc version zero = false 2024-03-28T16:13:56.528 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on no sortbitwise = false 2024-03-28T16:13:56.528 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on osd down out interval zero = false 2024-03-28T16:13:56.528 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on too few osds = false 2024-03-28T16:13:56.528 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-28T16:13:56.528 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-28T16:13:56.528 INFO:teuthology.orchestra.run.smithi007.stdout: mon_allow_pool_size_one = true 2024-03-28T16:13:56.528 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.528 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-28T16:13:56.528 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.528 INFO:teuthology.orchestra.run.smithi007.stdout: osd default data pool replay window = 5 2024-03-28T16:13:56.529 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.529 INFO:teuthology.orchestra.run.smithi007.stdout: mon allow pool delete = true 2024-03-28T16:13:56.529 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.529 INFO:teuthology.orchestra.run.smithi007.stdout: mon cluster log file level = debug 2024-03-28T16:13:56.529 INFO:teuthology.orchestra.run.smithi007.stdout: debug asserts on shutdown = true 2024-03-28T16:13:56.529 INFO:teuthology.orchestra.run.smithi007.stdout: mon health detail to clog = false 2024-03-28T16:13:56.529 INFO:teuthology.orchestra.run.smithi007.stdout: mon host = "172.21.15.7,[v2:172.21.15.7:3301,v1:172.21.15.7:6790],172.21.15.83" 2024-03-28T16:13:56.529 INFO:teuthology.orchestra.run.smithi007.stdout: mon client directed command retry = 5 2024-03-28T16:13:56.529 INFO:teuthology.orchestra.run.smithi007.stdout: mon election default strategy = 1 2024-03-28T16:13:56.529 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject socket failures = 5000 2024-03-28T16:13:56.529 INFO:teuthology.orchestra.run.smithi007.stdout: ms type = async 2024-03-28T16:13:56.529 INFO:teuthology.orchestra.run.smithi007.stdout: osd heartbeat grace = 80 2024-03-28T16:13:56.529 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:13:56.529 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.529 INFO:teuthology.orchestra.run.smithi007.stdout:[osd] 2024-03-28T16:13:56.530 INFO:teuthology.orchestra.run.smithi007.stdout: osd journal size = 100 2024-03-28T16:13:56.530 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.530 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub load threshold = 5.0 2024-03-28T16:13:56.530 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub max interval = 600 2024-03-28T16:13:56.530 INFO:teuthology.orchestra.run.smithi007.stdout: osd mclock profile = high_recovery_ops 2024-03-28T16:13:56.530 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.530 INFO:teuthology.orchestra.run.smithi007.stdout: osd recover clone overlap = true 2024-03-28T16:13:56.530 INFO:teuthology.orchestra.run.smithi007.stdout: osd recovery max chunk = 1048576 2024-03-28T16:13:56.530 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.530 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug shutdown = true 2024-03-28T16:13:56.530 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug op order = true 2024-03-28T16:13:56.530 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify stray on activate = true 2024-03-28T16:13:56.530 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.530 INFO:teuthology.orchestra.run.smithi007.stdout: osd open classes on start = true 2024-03-28T16:13:56.530 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug pg log writeout = true 2024-03-28T16:13:56.531 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.531 INFO:teuthology.orchestra.run.smithi007.stdout: osd deep scrub update digest min age = 30 2024-03-28T16:13:56.531 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.531 INFO:teuthology.orchestra.run.smithi007.stdout: osd map max advance = 10 2024-03-28T16:13:56.531 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.531 INFO:teuthology.orchestra.run.smithi007.stdout: journal zero on create = true 2024-03-28T16:13:56.531 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.531 INFO:teuthology.orchestra.run.smithi007.stdout: filestore ondisk finisher threads = 3 2024-03-28T16:13:56.531 INFO:teuthology.orchestra.run.smithi007.stdout: filestore apply finisher threads = 3 2024-03-28T16:13:56.531 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.531 INFO:teuthology.orchestra.run.smithi007.stdout: bdev debug aio = true 2024-03-28T16:13:56.531 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug misdirected ops = true 2024-03-28T16:13:56.531 INFO:teuthology.orchestra.run.smithi007.stdout: bdev async discard = True 2024-03-28T16:13:56.531 INFO:teuthology.orchestra.run.smithi007.stdout: bdev enable discard = True 2024-03-28T16:13:56.531 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore allocator = bitmap 2024-03-28T16:13:56.532 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore block size = 96636764160 2024-03-28T16:13:56.532 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore fsck on mount = True 2024-03-28T16:13:56.532 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore zero block detection = True 2024-03-28T16:13:56.532 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluefs = 1 2024-03-28T16:13:56.532 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluestore = 1 2024-03-28T16:13:56.532 INFO:teuthology.orchestra.run.smithi007.stdout: debug monc = 20 2024-03-28T16:13:56.532 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-03-28T16:13:56.532 INFO:teuthology.orchestra.run.smithi007.stdout: debug osd = 20 2024-03-28T16:13:56.532 INFO:teuthology.orchestra.run.smithi007.stdout: debug rocksdb = 10 2024-03-28T16:13:56.532 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd backfillfull_ratio = 0.85 2024-03-28T16:13:56.532 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd full ratio = 0.9 2024-03-28T16:13:56.532 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd nearfull ratio = 0.8 2024-03-28T16:13:56.532 INFO:teuthology.orchestra.run.smithi007.stdout: osd class default list = * 2024-03-28T16:13:56.532 INFO:teuthology.orchestra.run.smithi007.stdout: osd class load list = * 2024-03-28T16:13:56.532 INFO:teuthology.orchestra.run.smithi007.stdout: osd client watch timeout = 120 2024-03-28T16:13:56.533 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify cached snaps = True 2024-03-28T16:13:56.533 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify missing on start = True 2024-03-28T16:13:56.533 INFO:teuthology.orchestra.run.smithi007.stdout: osd failsafe full ratio = 0.95 2024-03-28T16:13:56.533 INFO:teuthology.orchestra.run.smithi007.stdout: osd fast shutdown = False 2024-03-28T16:13:56.533 INFO:teuthology.orchestra.run.smithi007.stdout: osd mclock override recovery settings = True 2024-03-28T16:13:56.533 INFO:teuthology.orchestra.run.smithi007.stdout: osd objectstore = bluestore 2024-03-28T16:13:56.533 INFO:teuthology.orchestra.run.smithi007.stdout: osd op queue = debug_random 2024-03-28T16:13:56.533 INFO:teuthology.orchestra.run.smithi007.stdout: osd op queue cut off = debug_random 2024-03-28T16:13:56.533 INFO:teuthology.orchestra.run.smithi007.stdout: osd shutdown pgref assert = True 2024-03-28T16:13:56.533 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.533 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr] 2024-03-28T16:13:56.533 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-03-28T16:13:56.533 INFO:teuthology.orchestra.run.smithi007.stdout: debug mgr = 20 2024-03-28T16:13:56.533 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2024-03-28T16:13:56.533 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2024-03-28T16:13:56.533 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2024-03-28T16:13:56.534 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2024-03-28T16:13:56.534 INFO:teuthology.orchestra.run.smithi007.stdout: mgr/telemetry/nag = false 2024-03-28T16:13:56.534 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.534 INFO:teuthology.orchestra.run.smithi007.stdout:[mon] 2024-03-28T16:13:56.534 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-03-28T16:13:56.534 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2024-03-28T16:13:56.534 INFO:teuthology.orchestra.run.smithi007.stdout: debug paxos = 20 2024-03-28T16:13:56.534 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2024-03-28T16:13:56.534 INFO:teuthology.orchestra.run.smithi007.stdout: mon data avail warn = 5 2024-03-28T16:13:56.534 INFO:teuthology.orchestra.run.smithi007.stdout: mon mgr mkfs grace = 240 2024-03-28T16:13:56.534 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2024-03-28T16:13:56.534 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd reporter subtree level = osd 2024-03-28T16:13:56.534 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd prime pg temp = true 2024-03-28T16:13:56.534 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2024-03-28T16:13:56.534 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.535 INFO:teuthology.orchestra.run.smithi007.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-28T16:13:56.535 INFO:teuthology.orchestra.run.smithi007.stdout: auth mon ticket ttl = 660 # 11m 2024-03-28T16:13:56.535 INFO:teuthology.orchestra.run.smithi007.stdout: auth service ticket ttl = 240 # 4m 2024-03-28T16:13:56.535 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.535 INFO:teuthology.orchestra.run.smithi007.stdout: # don't complain about insecure global_id in the test suite 2024-03-28T16:13:56.535 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-28T16:13:56.535 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-28T16:13:56.535 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.535 INFO:teuthology.orchestra.run.smithi007.stdout: # 1m isn't quite enough 2024-03-28T16:13:56.535 INFO:teuthology.orchestra.run.smithi007.stdout: mon_down_mkfs_grace = 2m 2024-03-28T16:13:56.535 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.535 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_filestore_osds = false 2024-03-28T16:13:56.535 INFO:teuthology.orchestra.run.smithi007.stdout: mon min osdmap epochs = 50 2024-03-28T16:13:56.535 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd crush smoke test = False 2024-03-28T16:13:56.535 INFO:teuthology.orchestra.run.smithi007.stdout: mon osdmap full prune interval = 2 2024-03-28T16:13:56.535 INFO:teuthology.orchestra.run.smithi007.stdout: mon osdmap full prune min = 15 2024-03-28T16:13:56.536 INFO:teuthology.orchestra.run.smithi007.stdout: mon osdmap full prune txsize = 2 2024-03-28T16:13:56.536 INFO:teuthology.orchestra.run.smithi007.stdout: mon scrub interval = 300 2024-03-28T16:13:56.536 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on pool no app = False 2024-03-28T16:13:56.536 INFO:teuthology.orchestra.run.smithi007.stdout: paxos service trim min = 10 2024-03-28T16:13:56.536 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:13:56.536 INFO:teuthology.orchestra.run.smithi007.stdout:[client] 2024-03-28T16:13:56.536 INFO:teuthology.orchestra.run.smithi007.stdout: rgw cache enabled = true 2024-03-28T16:13:56.536 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable ops log = true 2024-03-28T16:13:56.536 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable usage log = true 2024-03-28T16:13:56.536 INFO:teuthology.orchestra.run.smithi007.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-28T16:13:56.536 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-28T16:13:56.536 INFO:teuthology.orchestra.run.smithi007.stdout: debug monc = 20 2024-03-28T16:13:56.536 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-03-28T16:13:56.536 INFO:teuthology.orchestra.run.smithi007.stdout: debug objecter = 20 2024-03-28T16:13:56.536 INFO:teuthology.orchestra.run.smithi007.stdout: debug rados = 20 2024-03-28T16:13:56.536 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.a] 2024-03-28T16:13:56.537 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.c] 2024-03-28T16:13:56.537 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.b] 2024-03-28T16:13:56.537 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2024-03-28T16:13:56.537 INFO:teuthology.orchestra.run.smithi083.stdout: chdir = "" 2024-03-28T16:13:56.537 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-28T16:13:56.537 INFO:teuthology.orchestra.run.smithi083.stdout: auth supported = cephx 2024-03-28T16:13:56.537 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.537 INFO:teuthology.orchestra.run.smithi083.stdout: filestore xattr use omap = true 2024-03-28T16:13:56.538 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.538 INFO:teuthology.orchestra.run.smithi083.stdout: mon clock drift allowed = 1.000 2024-03-28T16:13:56.538 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.538 INFO:teuthology.orchestra.run.smithi083.stdout: osd crush chooseleaf type = 0 2024-03-28T16:13:56.538 INFO:teuthology.orchestra.run.smithi083.stdout: auth debug = true 2024-03-28T16:13:56.538 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.538 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on old message = true 2024-03-28T16:13:56.538 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on bug = true 2024-03-28T16:13:56.538 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.538 INFO:teuthology.orchestra.run.smithi083.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-28T16:13:56.538 INFO:teuthology.orchestra.run.smithi083.stdout: mon pg warn max object skew = 0 2024-03-28T16:13:56.538 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.538 INFO:teuthology.orchestra.run.smithi083.stdout: # disable pg_autoscaler by default for new pools 2024-03-28T16:13:56.538 INFO:teuthology.orchestra.run.smithi083.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-28T16:13:56.538 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.539 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default size = 2 2024-03-28T16:13:56.539 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.539 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow primary affinity = true 2024-03-28T16:13:56.539 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow pg remap = true 2024-03-28T16:13:56.539 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on legacy crush tunables = false 2024-03-28T16:13:56.539 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on crush straw calc version zero = false 2024-03-28T16:13:56.539 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on no sortbitwise = false 2024-03-28T16:13:56.539 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on osd down out interval zero = false 2024-03-28T16:13:56.539 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on too few osds = false 2024-03-28T16:13:56.539 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-28T16:13:56.539 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-28T16:13:56.539 INFO:teuthology.orchestra.run.smithi083.stdout: mon_allow_pool_size_one = true 2024-03-28T16:13:56.539 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.539 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-28T16:13:56.539 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.540 INFO:teuthology.orchestra.run.smithi083.stdout: osd default data pool replay window = 5 2024-03-28T16:13:56.540 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.540 INFO:teuthology.orchestra.run.smithi083.stdout: mon allow pool delete = true 2024-03-28T16:13:56.540 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.540 INFO:teuthology.orchestra.run.smithi083.stdout: mon cluster log file level = debug 2024-03-28T16:13:56.540 INFO:teuthology.orchestra.run.smithi083.stdout: debug asserts on shutdown = true 2024-03-28T16:13:56.540 INFO:teuthology.orchestra.run.smithi083.stdout: mon health detail to clog = false 2024-03-28T16:13:56.540 INFO:teuthology.orchestra.run.smithi083.stdout: mon host = "172.21.15.7,[v2:172.21.15.7:3301,v1:172.21.15.7:6790],172.21.15.83" 2024-03-28T16:13:56.540 INFO:teuthology.orchestra.run.smithi083.stdout: mon client directed command retry = 5 2024-03-28T16:13:56.540 INFO:teuthology.orchestra.run.smithi083.stdout: mon election default strategy = 1 2024-03-28T16:13:56.540 INFO:teuthology.orchestra.run.smithi083.stdout: ms inject socket failures = 5000 2024-03-28T16:13:56.540 INFO:teuthology.orchestra.run.smithi083.stdout: ms type = async 2024-03-28T16:13:56.540 INFO:teuthology.orchestra.run.smithi083.stdout: osd heartbeat grace = 80 2024-03-28T16:13:56.540 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:13:56.540 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.541 INFO:teuthology.orchestra.run.smithi083.stdout:[osd] 2024-03-28T16:13:56.541 INFO:teuthology.orchestra.run.smithi083.stdout: osd journal size = 100 2024-03-28T16:13:56.541 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.541 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub load threshold = 5.0 2024-03-28T16:13:56.541 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub max interval = 600 2024-03-28T16:13:56.541 INFO:teuthology.orchestra.run.smithi083.stdout: osd mclock profile = high_recovery_ops 2024-03-28T16:13:56.541 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.541 INFO:teuthology.orchestra.run.smithi083.stdout: osd recover clone overlap = true 2024-03-28T16:13:56.541 INFO:teuthology.orchestra.run.smithi083.stdout: osd recovery max chunk = 1048576 2024-03-28T16:13:56.541 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.541 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug shutdown = true 2024-03-28T16:13:56.541 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug op order = true 2024-03-28T16:13:56.541 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify stray on activate = true 2024-03-28T16:13:56.541 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.541 INFO:teuthology.orchestra.run.smithi083.stdout: osd open classes on start = true 2024-03-28T16:13:56.542 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug pg log writeout = true 2024-03-28T16:13:56.542 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.542 INFO:teuthology.orchestra.run.smithi083.stdout: osd deep scrub update digest min age = 30 2024-03-28T16:13:56.542 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.542 INFO:teuthology.orchestra.run.smithi083.stdout: osd map max advance = 10 2024-03-28T16:13:56.542 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.542 INFO:teuthology.orchestra.run.smithi083.stdout: journal zero on create = true 2024-03-28T16:13:56.542 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.542 INFO:teuthology.orchestra.run.smithi083.stdout: filestore ondisk finisher threads = 3 2024-03-28T16:13:56.542 INFO:teuthology.orchestra.run.smithi083.stdout: filestore apply finisher threads = 3 2024-03-28T16:13:56.542 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.542 INFO:teuthology.orchestra.run.smithi083.stdout: bdev debug aio = true 2024-03-28T16:13:56.542 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug misdirected ops = true 2024-03-28T16:13:56.542 INFO:teuthology.orchestra.run.smithi083.stdout: bdev async discard = True 2024-03-28T16:13:56.542 INFO:teuthology.orchestra.run.smithi083.stdout: bdev enable discard = True 2024-03-28T16:13:56.543 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore allocator = bitmap 2024-03-28T16:13:56.543 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore block size = 96636764160 2024-03-28T16:13:56.543 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore fsck on mount = True 2024-03-28T16:13:56.543 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore zero block detection = True 2024-03-28T16:13:56.543 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluefs = 1 2024-03-28T16:13:56.543 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluestore = 1 2024-03-28T16:13:56.543 INFO:teuthology.orchestra.run.smithi083.stdout: debug monc = 20 2024-03-28T16:13:56.543 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-28T16:13:56.543 INFO:teuthology.orchestra.run.smithi083.stdout: debug osd = 20 2024-03-28T16:13:56.543 INFO:teuthology.orchestra.run.smithi083.stdout: debug rocksdb = 10 2024-03-28T16:13:56.543 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd backfillfull_ratio = 0.85 2024-03-28T16:13:56.543 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd full ratio = 0.9 2024-03-28T16:13:56.543 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd nearfull ratio = 0.8 2024-03-28T16:13:56.543 INFO:teuthology.orchestra.run.smithi083.stdout: osd class default list = * 2024-03-28T16:13:56.543 INFO:teuthology.orchestra.run.smithi083.stdout: osd class load list = * 2024-03-28T16:13:56.544 INFO:teuthology.orchestra.run.smithi083.stdout: osd client watch timeout = 120 2024-03-28T16:13:56.544 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify cached snaps = True 2024-03-28T16:13:56.544 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify missing on start = True 2024-03-28T16:13:56.544 INFO:teuthology.orchestra.run.smithi083.stdout: osd failsafe full ratio = 0.95 2024-03-28T16:13:56.544 INFO:teuthology.orchestra.run.smithi083.stdout: osd fast shutdown = False 2024-03-28T16:13:56.544 INFO:teuthology.orchestra.run.smithi083.stdout: osd mclock override recovery settings = True 2024-03-28T16:13:56.544 INFO:teuthology.orchestra.run.smithi083.stdout: osd objectstore = bluestore 2024-03-28T16:13:56.544 INFO:teuthology.orchestra.run.smithi083.stdout: osd op queue = debug_random 2024-03-28T16:13:56.544 INFO:teuthology.orchestra.run.smithi083.stdout: osd op queue cut off = debug_random 2024-03-28T16:13:56.544 INFO:teuthology.orchestra.run.smithi083.stdout: osd shutdown pgref assert = True 2024-03-28T16:13:56.544 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.544 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr] 2024-03-28T16:13:56.544 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-28T16:13:56.544 INFO:teuthology.orchestra.run.smithi083.stdout: debug mgr = 20 2024-03-28T16:13:56.544 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-03-28T16:13:56.544 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-03-28T16:13:56.545 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-03-28T16:13:56.545 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-03-28T16:13:56.545 INFO:teuthology.orchestra.run.smithi083.stdout: mgr/telemetry/nag = false 2024-03-28T16:13:56.545 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.545 INFO:teuthology.orchestra.run.smithi083.stdout:[mon] 2024-03-28T16:13:56.545 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-28T16:13:56.545 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-03-28T16:13:56.545 INFO:teuthology.orchestra.run.smithi083.stdout: debug paxos = 20 2024-03-28T16:13:56.545 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-03-28T16:13:56.545 INFO:teuthology.orchestra.run.smithi083.stdout: mon data avail warn = 5 2024-03-28T16:13:56.545 INFO:teuthology.orchestra.run.smithi083.stdout: mon mgr mkfs grace = 240 2024-03-28T16:13:56.545 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-03-28T16:13:56.545 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd reporter subtree level = osd 2024-03-28T16:13:56.545 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd prime pg temp = true 2024-03-28T16:13:56.546 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-03-28T16:13:56.546 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.546 INFO:teuthology.orchestra.run.smithi083.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-28T16:13:56.546 INFO:teuthology.orchestra.run.smithi083.stdout: auth mon ticket ttl = 660 # 11m 2024-03-28T16:13:56.546 INFO:teuthology.orchestra.run.smithi083.stdout: auth service ticket ttl = 240 # 4m 2024-03-28T16:13:56.546 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.546 INFO:teuthology.orchestra.run.smithi083.stdout: # don't complain about insecure global_id in the test suite 2024-03-28T16:13:56.546 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-28T16:13:56.546 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-28T16:13:56.546 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.546 INFO:teuthology.orchestra.run.smithi083.stdout: # 1m isn't quite enough 2024-03-28T16:13:56.546 INFO:teuthology.orchestra.run.smithi083.stdout: mon_down_mkfs_grace = 2m 2024-03-28T16:13:56.546 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.546 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_filestore_osds = false 2024-03-28T16:13:56.547 INFO:teuthology.orchestra.run.smithi083.stdout: mon min osdmap epochs = 50 2024-03-28T16:13:56.547 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd crush smoke test = False 2024-03-28T16:13:56.547 INFO:teuthology.orchestra.run.smithi083.stdout: mon osdmap full prune interval = 2 2024-03-28T16:13:56.547 INFO:teuthology.orchestra.run.smithi083.stdout: mon osdmap full prune min = 15 2024-03-28T16:13:56.547 INFO:teuthology.orchestra.run.smithi083.stdout: mon osdmap full prune txsize = 2 2024-03-28T16:13:56.547 INFO:teuthology.orchestra.run.smithi083.stdout: mon scrub interval = 300 2024-03-28T16:13:56.547 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on pool no app = False 2024-03-28T16:13:56.547 INFO:teuthology.orchestra.run.smithi083.stdout: paxos service trim min = 10 2024-03-28T16:13:56.547 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-28T16:13:56.547 INFO:teuthology.orchestra.run.smithi083.stdout:[client] 2024-03-28T16:13:56.547 INFO:teuthology.orchestra.run.smithi083.stdout: rgw cache enabled = true 2024-03-28T16:13:56.547 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable ops log = true 2024-03-28T16:13:56.547 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable usage log = true 2024-03-28T16:13:56.547 INFO:teuthology.orchestra.run.smithi083.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-28T16:13:56.547 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-28T16:13:56.548 INFO:teuthology.orchestra.run.smithi083.stdout: debug monc = 20 2024-03-28T16:13:56.548 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-28T16:13:56.548 INFO:teuthology.orchestra.run.smithi083.stdout: debug objecter = 20 2024-03-28T16:13:56.548 INFO:teuthology.orchestra.run.smithi083.stdout: debug rados = 20 2024-03-28T16:13:56.548 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.a] 2024-03-28T16:13:56.548 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.c] 2024-03-28T16:13:56.548 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.b] 2024-03-28T16:13:56.549 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-28T16:13:56.549 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-28T16:13:56.644 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-28T16:13:56.644 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:13:56.644 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-28T16:13:56.659 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:13:56.659 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-28T16:13:56.713 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi007.front.sepia.ceph.com 2024-03-28T16:13:56.713 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:13:56.713 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-28T16:13:56.713 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-28T16:13:56.791 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:13:56.791 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-28T16:13:56.845 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi083.front.sepia.ceph.com 2024-03-28T16:13:56.845 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-28T16:13:56.845 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-28T16:13:56.845 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-28T16:13:56.885 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-28T16:13:56.886 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-28T16:13:56.941 INFO:tasks.ceph:Setting up mon nodes... 2024-03-28T16:13:56.941 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-28T16:13:56.942 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-28T16:13:56.995 INFO:teuthology.orchestra.run.smithi007.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-28T16:13:56.998 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-28T16:13:57.065 INFO:teuthology.orchestra.run.smithi083.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-28T16:13:57.067 INFO:tasks.ceph:Setting up mds nodes... 2024-03-28T16:13:57.068 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-28T16:13:57.068 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-28T16:13:57.109 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-28T16:13:57.124 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-28T16:13:57.166 INFO:teuthology.orchestra.run.smithi083.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-28T16:13:57.182 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-28T16:13:57.182 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi083.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-28T16:13:57.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-28T16:13:57.209 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T16:13:57.209 INFO:tasks.ceph:role: osd.0 2024-03-28T16:13:57.209 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-28T16:13:57.209 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-28T16:13:57.274 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T16:13:57.275 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T16:13:57.275 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T16:13:57.275 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T16:13:57.275 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T16:13:57.275 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-03-28T16:13:57.275 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T16:13:57.275 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T16:13:57.275 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T16:13:57.275 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T16:13:57.307 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-03-28T16:13:57.308 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-03-28T16:13:57.308 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-28T16:13:57.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-28T16:13:57.450 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-28T16:13:57.517 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T16:13:57.518 INFO:tasks.ceph:role: osd.1 2024-03-28T16:13:57.518 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-28T16:13:57.518 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-28T16:13:57.584 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T16:13:57.584 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T16:13:57.584 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T16:13:57.584 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T16:13:57.584 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T16:13:57.584 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-03-28T16:13:57.584 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T16:13:57.584 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T16:13:57.584 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T16:13:57.584 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T16:13:57.616 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-03-28T16:13:57.618 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-03-28T16:13:57.618 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-28T16:13:57.689 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-28T16:13:57.760 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-28T16:13:57.827 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T16:13:57.827 INFO:tasks.ceph:role: osd.2 2024-03-28T16:13:57.827 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-28T16:13:57.827 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-28T16:13:57.892 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T16:13:57.893 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T16:13:57.893 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T16:13:57.893 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T16:13:57.893 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T16:13:57.893 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-03-28T16:13:57.893 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T16:13:57.893 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T16:13:57.894 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T16:13:57.894 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T16:13:57.924 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-03-28T16:13:57.926 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-03-28T16:13:57.926 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-28T16:13:57.997 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-28T16:13:58.067 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-28T16:13:58.133 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-28T16:13:58.133 INFO:tasks.ceph:role: osd.3 2024-03-28T16:13:58.133 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-28T16:13:58.134 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-28T16:13:58.200 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T16:13:58.200 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T16:13:58.200 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T16:13:58.200 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T16:13:58.200 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T16:13:58.200 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-03-28T16:13:58.200 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T16:13:58.200 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T16:13:58.201 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T16:13:58.201 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T16:13:58.232 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-03-28T16:13:58.234 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-03-28T16:13:58.234 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-28T16:13:58.309 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-28T16:13:58.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T16:13:58.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:13:58.477+0000 7fb57d637640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-28T16:13:58.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:13:58.477+0000 7fb57d637640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-28T16:13:58.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:13:58.477+0000 7fb57d637640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-28T16:13:58.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:13:58.477+0000 7fb57d637640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-28T16:13:58.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:13:58.477+0000 7fb57d637640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-28T16:13:58.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:13:58.477+0000 7fb57d637640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-28T16:14:01.316 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-28T16:14:01.348 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T16:14:01.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:01.438+0000 7f0f04b28640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-28T16:14:01.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:01.438+0000 7f0f04b28640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-28T16:14:01.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:01.438+0000 7f0f04b28640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-28T16:14:01.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:01.438+0000 7f0f04b28640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-28T16:14:01.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:01.438+0000 7f0f04b28640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-28T16:14:01.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:01.439+0000 7f0f04b28640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-28T16:14:04.277 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-28T16:14:04.308 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T16:14:04.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:04.399+0000 7fdaf3248640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-28T16:14:04.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:04.399+0000 7fdaf3248640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-28T16:14:04.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:04.399+0000 7fdaf3248640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-28T16:14:04.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:04.399+0000 7fdaf3248640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-28T16:14:04.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:04.399+0000 7fdaf3248640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-28T16:14:04.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:04.400+0000 7fdaf3248640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-28T16:14:07.243 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-28T16:14:07.274 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T16:14:07.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:07.364+0000 7f05c3947640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-28T16:14:07.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:07.364+0000 7f05c3947640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-28T16:14:07.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:07.364+0000 7f05c3947640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-28T16:14:07.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:07.364+0000 7f05c3947640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-28T16:14:07.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:07.364+0000 7f05c3947640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-28T16:14:07.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:07.365+0000 7f05c3947640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-28T16:14:10.204 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-28T16:14:10.234 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-28T16:14:10.267 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T16:14:10.267 INFO:tasks.ceph:role: osd.4 2024-03-28T16:14:10.267 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-28T16:14:10.267 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-28T16:14:10.345 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T16:14:10.345 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T16:14:10.345 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T16:14:10.345 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T16:14:10.345 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T16:14:10.345 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-03-28T16:14:10.345 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T16:14:10.345 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T16:14:10.345 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T16:14:10.345 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T16:14:10.855 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-03-28T16:14:10.858 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-03-28T16:14:10.858 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-28T16:14:11.001 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-28T16:14:11.037 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-28T16:14:11.108 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T16:14:11.108 INFO:tasks.ceph:role: osd.5 2024-03-28T16:14:11.108 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-28T16:14:11.109 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-28T16:14:11.185 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T16:14:11.185 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T16:14:11.185 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T16:14:11.186 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T16:14:11.186 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T16:14:11.186 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-03-28T16:14:11.186 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T16:14:11.186 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T16:14:11.186 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T16:14:11.186 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T16:14:11.637 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-03-28T16:14:11.639 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-03-28T16:14:11.639 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-28T16:14:11.736 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-28T16:14:11.769 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-28T16:14:11.840 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T16:14:11.840 INFO:tasks.ceph:role: osd.6 2024-03-28T16:14:11.840 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-28T16:14:11.840 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-28T16:14:11.916 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T16:14:11.916 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T16:14:11.916 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T16:14:11.916 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T16:14:11.916 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T16:14:11.916 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-03-28T16:14:11.916 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T16:14:11.916 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T16:14:11.916 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T16:14:11.916 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T16:14:12.406 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-03-28T16:14:12.409 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-03-28T16:14:12.409 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-28T16:14:12.563 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-28T16:14:12.598 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-28T16:14:12.667 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-28T16:14:12.668 INFO:tasks.ceph:role: osd.7 2024-03-28T16:14:12.668 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-28T16:14:12.668 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-28T16:14:12.744 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T16:14:12.745 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T16:14:12.745 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T16:14:12.745 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T16:14:12.745 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T16:14:12.745 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-03-28T16:14:12.745 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T16:14:12.745 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T16:14:12.745 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T16:14:12.745 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T16:14:13.250 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-03-28T16:14:13.252 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-03-28T16:14:13.252 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-28T16:14:13.374 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-28T16:14:13.411 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T16:14:13.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:13.531+0000 7fc91f8a6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-28T16:14:13.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:13.531+0000 7fc91f8a6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-28T16:14:13.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:13.531+0000 7fc91f8a6640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-28T16:14:13.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:13.531+0000 7fc91f8a6640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-28T16:14:13.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:13.531+0000 7fc91f8a6640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-28T16:14:13.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:13.532+0000 7fc91f8a6640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-28T16:14:16.385 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-28T16:14:16.416 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T16:14:16.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:16.506+0000 7f94ca09d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-28T16:14:16.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:16.506+0000 7f94ca09d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-28T16:14:16.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:16.506+0000 7f94ca09d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-28T16:14:16.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:16.507+0000 7f94ca09d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-28T16:14:16.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:16.507+0000 7f94ca09d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-28T16:14:16.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:16.507+0000 7f94ca09d640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-28T16:14:19.348 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-28T16:14:19.383 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T16:14:19.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:19.477+0000 7f8ee8e15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-28T16:14:19.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:19.478+0000 7f8ee8e15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-28T16:14:19.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:19.478+0000 7f8ee8e15640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-28T16:14:19.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:19.478+0000 7f8ee8e15640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-28T16:14:19.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:19.478+0000 7f8ee8e15640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-28T16:14:19.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:19.478+0000 7f8ee8e15640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-28T16:14:22.322 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-28T16:14:22.353 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T16:14:22.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:22.443+0000 7f3e8a547640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-28T16:14:22.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:22.443+0000 7f3e8a547640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-28T16:14:22.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:22.443+0000 7f3e8a547640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-28T16:14:22.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:22.443+0000 7f3e8a547640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-28T16:14:22.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:22.443+0000 7f3e8a547640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-28T16:14:22.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:22.444+0000 7f3e8a547640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-28T16:14:25.286 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-28T16:14:25.318 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-28T16:14:25.318 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:14:25.318 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-28T16:14:25.350 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:14:25.351 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-28T16:14:25.422 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:14:25.422 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-28T16:14:25.491 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:14:25.492 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-28T16:14:25.561 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:14:25.561 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-28T16:14:25.629 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-28T16:14:25.629 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-28T16:14:25.661 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-28T16:14:25.661 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-28T16:14:25.732 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-28T16:14:25.732 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-28T16:14:25.802 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-28T16:14:25.802 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-28T16:14:25.871 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-28T16:14:25.871 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-28T16:14:25.940 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:14:25.940 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-28T16:14:25.958 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-28T16:14:25.958 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-28T16:14:25.996 INFO:tasks.ceph:Adding keys to all mons... 2024-03-28T16:14:25.996 DEBUG:teuthology.orchestra.run.smithi007:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-28T16:14:26.000 DEBUG:teuthology.orchestra.run.smithi083:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-28T16:14:26.039 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr.y] 2024-03-28T16:14:26.039 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBElwVmVhZkOxAAFuqte+E6iVlUag0iuThvIA== 2024-03-28T16:14:26.039 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.0] 2024-03-28T16:14:26.039 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBGlwVm9vaGHBAAnrkIok0qOkhre+IIyOnB1A== 2024-03-28T16:14:26.039 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.1] 2024-03-28T16:14:26.039 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBJlwVmdYk+GhAA9z4l9ebmLNeQlEV0+jn8TQ== 2024-03-28T16:14:26.039 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.2] 2024-03-28T16:14:26.039 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBMlwVm5InqFxAALtFtH9/fvh1cQV3FTtQ+nQ== 2024-03-28T16:14:26.040 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.3] 2024-03-28T16:14:26.040 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBPlwVmc0zRFRAA+xYEqn/HD24eBgAUe1BUEg== 2024-03-28T16:14:26.040 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr.x] 2024-03-28T16:14:26.040 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBFlwVmzH7wAxAA7zORGEKttz9AbsqUjzTSOA== 2024-03-28T16:14:26.040 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.4] 2024-03-28T16:14:26.040 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBVlwVmgjfGHxAAsq7/ifOa9YWBek/zyOraFg== 2024-03-28T16:14:26.040 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.5] 2024-03-28T16:14:26.040 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBYlwVmna9JHhAAGr3kdDewaHI2gT+5DNAnbQ== 2024-03-28T16:14:26.040 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.6] 2024-03-28T16:14:26.040 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBblwVm3yKTHBAA2I1p5qtYTVCaDsjz6v1k1A== 2024-03-28T16:14:26.040 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.7] 2024-03-28T16:14:26.040 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBelwVmxD2GGhAAsGOe9CA5Z/JJrsRd0YbI3w== 2024-03-28T16:14:26.040 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2024-03-28T16:14:26.040 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBFlwVmOCeVBhAACW8mRRGqQzspFwefoLi9OQ== 2024-03-28T16:14:26.041 INFO:teuthology.orchestra.run.smithi007.stdout:[client.1] 2024-03-28T16:14:26.041 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBFlwVm8nj9CRAATjKTXkyw6gQuFIdNzrfNvA== 2024-03-28T16:14:26.062 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr.y] 2024-03-28T16:14:26.062 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBElwVmVhZkOxAAFuqte+E6iVlUag0iuThvIA== 2024-03-28T16:14:26.062 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.0] 2024-03-28T16:14:26.062 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBGlwVm9vaGHBAAnrkIok0qOkhre+IIyOnB1A== 2024-03-28T16:14:26.062 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.1] 2024-03-28T16:14:26.062 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBJlwVmdYk+GhAA9z4l9ebmLNeQlEV0+jn8TQ== 2024-03-28T16:14:26.062 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.2] 2024-03-28T16:14:26.062 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBMlwVm5InqFxAALtFtH9/fvh1cQV3FTtQ+nQ== 2024-03-28T16:14:26.062 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.3] 2024-03-28T16:14:26.063 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBPlwVmc0zRFRAA+xYEqn/HD24eBgAUe1BUEg== 2024-03-28T16:14:26.063 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr.x] 2024-03-28T16:14:26.063 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBFlwVmzH7wAxAA7zORGEKttz9AbsqUjzTSOA== 2024-03-28T16:14:26.063 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.4] 2024-03-28T16:14:26.064 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBVlwVmgjfGHxAAsq7/ifOa9YWBek/zyOraFg== 2024-03-28T16:14:26.064 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.5] 2024-03-28T16:14:26.064 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBYlwVmna9JHhAAGr3kdDewaHI2gT+5DNAnbQ== 2024-03-28T16:14:26.064 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.6] 2024-03-28T16:14:26.064 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBblwVm3yKTHBAA2I1p5qtYTVCaDsjz6v1k1A== 2024-03-28T16:14:26.064 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.7] 2024-03-28T16:14:26.064 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBelwVmxD2GGhAAsGOe9CA5Z/JJrsRd0YbI3w== 2024-03-28T16:14:26.064 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2024-03-28T16:14:26.064 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBFlwVmOCeVBhAACW8mRRGqQzspFwefoLi9OQ== 2024-03-28T16:14:26.064 INFO:teuthology.orchestra.run.smithi083.stdout:[client.1] 2024-03-28T16:14:26.064 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBFlwVm8nj9CRAATjKTXkyw6gQuFIdNzrfNvA== 2024-03-28T16:14:26.065 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-28T16:14:26.084 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-28T16:14:26.163 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T16:14:26.165 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T16:14:26.217 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T16:14:26.219 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T16:14:26.270 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T16:14:26.272 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T16:14:26.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T16:14:26.325 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T16:14:26.376 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-28T16:14:26.379 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-28T16:14:26.429 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T16:14:26.432 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T16:14:26.483 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T16:14:26.485 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T16:14:26.536 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T16:14:26.539 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T16:14:26.590 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T16:14:26.592 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T16:14:26.643 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-28T16:14:26.646 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-28T16:14:26.700 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-28T16:14:26.702 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-28T16:14:26.755 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-28T16:14:26.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-28T16:14:26.782 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-28T16:14:26.878 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-28T16:14:26.904 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-28T16:14:26.971 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-28T16:14:27.068 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-28T16:14:27.095 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-28T16:14:27.122 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-28T16:14:27.220 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-28T16:14:27.246 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-28T16:14:27.248 DEBUG:teuthology.orchestra.run.smithi083:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-28T16:14:27.301 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-28T16:14:27.302 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-28T16:14:27.302 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-28T16:14:27.302 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-28T16:14:27.304 INFO:tasks.ceph.mon.a:Started 2024-03-28T16:14:27.304 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-28T16:14:27.304 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-28T16:14:27.304 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-28T16:14:27.306 INFO:tasks.ceph.mon.c:Started 2024-03-28T16:14:27.306 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-28T16:14:27.307 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-28T16:14:27.307 DEBUG:teuthology.orchestra.run.smithi083:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-28T16:14:27.344 INFO:tasks.ceph.mon.b:Started 2024-03-28T16:14:27.344 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-28T16:14:27.344 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-28T16:14:27.345 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-28T16:14:27.347 INFO:tasks.ceph.mgr.y:Started 2024-03-28T16:14:27.347 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-28T16:14:27.348 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-28T16:14:27.349 INFO:tasks.ceph.mgr.x:Started 2024-03-28T16:14:27.349 INFO:tasks.ceph:Setting crush tunables to default 2024-03-28T16:14:27.350 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd crush tunables default 2024-03-28T16:14:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.450+0000 7fe6c5c56640 10 monclient: get_monmap_and_config 2024-03-28T16:14:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.450+0000 7fe6c5c56640 10 monclient: build_initial_monmap 2024-03-28T16:14:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.450+0000 7fe6c5c56640 10 monclient: monmap: 2024-03-28T16:14:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:14:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:14:27.451545+0000 2024-03-28T16:14:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:14:27.451545+0000 2024-03-28T16:14:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:14:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:14:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:14:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:14:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:14:27.452 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:14:27.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.451+0000 7fe6c5c56640 1 Processor -- start 2024-03-28T16:14:27.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.451+0000 7fe6c5c56640 1 -- start start 2024-03-28T16:14:27.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.451+0000 7fe6c5c56640 10 monclient: init 2024-03-28T16:14:27.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6c5c56640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:27.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6c5c56640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-03-28T16:14:27.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6c5c56640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe6c0071920 0x7fe6c013e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:27.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6c5c56640 10 monclient(hunting): picked mon.noname-c con 0x7fe6c0071920 addr v2:172.21.15.83:3300/0 2024-03-28T16:14:27.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6c5c56640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe6c00a89a0 0x7fe6c013ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:27.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6c5c56640 10 monclient(hunting): picked mon.noname-a con 0x7fe6c00a89a0 addr v2:172.21.15.7:3300/0 2024-03-28T16:14:27.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6c5c56640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6c00b1890 0x7fe6c00b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:27.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6c5c56640 10 monclient(hunting): picked mon.noname-b con 0x7fe6c00b1890 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:27.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6c5c56640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:27.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6c5c56640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6c00aa480 con 0x7fe6c00a89a0 2024-03-28T16:14:27.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6c5c56640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:27.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6c5c56640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6c0071d00 con 0x7fe6c0071920 2024-03-28T16:14:27.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6c5c56640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:27.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6c5c56640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c00a8ec0 con 0x7fe6c00b1890 2024-03-28T16:14:27.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6c5c56640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:27.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6c5c56640 10 monclient(hunting): authenticate will time out at 1334.593228s 2024-03-28T16:14:27.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6b6ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe6c00a89a0 msgr2=0x7fe6c013ed70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T16:14:27.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6bffff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6c00b1890 msgr2=0x7fe6c00b1d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T16:14:27.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6bffff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6c00b1890 0x7fe6c00b1d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T16:14:27.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6b6ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe6c00a89a0 0x7fe6c013ed70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T16:14:27.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6bf7fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe6c0071920 msgr2=0x7fe6c013e830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T16:14:27.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.452+0000 7fe6bf7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe6c0071920 0x7fe6c013e830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T16:14:27.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.653+0000 7fe6bffff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6c00b1890 msgr2=0x7fe6c00b1d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T16:14:27.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.653+0000 7fe6b6ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe6c00a89a0 msgr2=0x7fe6c013ed70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T16:14:27.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.653+0000 7fe6bffff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6c00b1890 0x7fe6c00b1d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T16:14:27.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.653+0000 7fe6b6ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe6c00a89a0 0x7fe6c013ed70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T16:14:27.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.653+0000 7fe6bf7fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe6c0071920 msgr2=0x7fe6c013e830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T16:14:27.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:27.653+0000 7fe6bf7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe6c0071920 0x7fe6c013e830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T16:14:28.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:28.054+0000 7fe6bffff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6c00b1890 msgr2=0x7fe6c00b1d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T16:14:28.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:28.054+0000 7fe6b6ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe6c00a89a0 msgr2=0x7fe6c013ed70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T16:14:28.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:28.054+0000 7fe6bffff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6c00b1890 0x7fe6c00b1d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T16:14:28.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:28.054+0000 7fe6b6ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe6c00a89a0 0x7fe6c013ed70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T16:14:28.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:28.054+0000 7fe6bf7fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe6c0071920 msgr2=0x7fe6c013e830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T16:14:28.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:28.054+0000 7fe6bf7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe6c0071920 0x7fe6c013e830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T16:14:28.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:28.855+0000 7fe6bffff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6c00b1890 msgr2=0x7fe6c00b1d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T16:14:28.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:28.855+0000 7fe6b6ffd640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe6c00a89a0 msgr2=0x7fe6c013ed70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T16:14:28.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:28.855+0000 7fe6b6ffd640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe6c00a89a0 0x7fe6c013ed70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T16:14:28.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:28.855+0000 7fe6bffff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6c00b1890 0x7fe6c00b1d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T16:14:28.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:28.855+0000 7fe6bf7fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe6c0071920 msgr2=0x7fe6c013e830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T16:14:28.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:28.855+0000 7fe6bf7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe6c0071920 0x7fe6c013e830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T16:14:30.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:30.452+0000 7fe6bdffb640 10 monclient(hunting): tick 2024-03-28T16:14:30.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:30.452+0000 7fe6bdffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T16:14:30.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:30.452+0000 7fe6bdffb640 1 monclient(hunting): continuing hunt 2024-03-28T16:14:30.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:30.452+0000 7fe6bdffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T16:14:30.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:30.452+0000 7fe6bdffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6c00b1890 msgr2=0x7fe6c00b1d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T16:14:30.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:30.452+0000 7fe6bdffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6c00b1890 0x7fe6c00b1d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:30.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:30.452+0000 7fe6bdffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe6c0071920 msgr2=0x7fe6c013e830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T16:14:30.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:30.452+0000 7fe6bdffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe6c0071920 0x7fe6c013e830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:30.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:30.452+0000 7fe6bdffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe6c00a89a0 msgr2=0x7fe6c013ed70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T16:14:30.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:30.452+0000 7fe6bdffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe6c00a89a0 0x7fe6c013ed70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:30.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:30.452+0000 7fe6bdffb640 10 monclient: _add_conns ranks=[0,1] 2024-03-28T16:14:30.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:30.452+0000 7fe6bdffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe6a00036e0 addr v1:172.21.15.7:6789/0 2024-03-28T16:14:30.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:30.452+0000 7fe6bdffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6a0007390 addr v1:172.21.15.83:6789/0 2024-03-28T16:14:30.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:30.452+0000 7fe6bdffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c00aa660 con 0x7fe6a00036e0 2024-03-28T16:14:30.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:30.452+0000 7fe6bdffb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a000c0c0 con 0x7fe6a0007390 2024-03-28T16:14:30.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:30.452+0000 7fe6bdffb640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:30.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:30.452+0000 7fe6bf7fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe6a00036e0 legacy=0x7fe6a0005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T16:14:30.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:30.452+0000 7fe6b6ffd640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe6a0007390 legacy=0x7fe6a0009850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T16:14:30.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:30.653+0000 7fe6bf7fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe6a00036e0 legacy=0x7fe6a0005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T16:14:30.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:30.653+0000 7fe6b6ffd640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe6a0007390 legacy=0x7fe6a0009850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T16:14:31.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:31.054+0000 7fe6bf7fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe6a00036e0 legacy=0x7fe6a0005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T16:14:31.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:31.054+0000 7fe6b6ffd640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe6a0007390 legacy=0x7fe6a0009850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T16:14:31.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:31.855+0000 7fe6bf7fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe6a00036e0 legacy=0x7fe6a0005b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T16:14:31.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:31.855+0000 7fe6b6ffd640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe6a0007390 legacy=0x7fe6a0009850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T16:14:33.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bdffb640 10 monclient(hunting): tick 2024-03-28T16:14:33.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bdffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T16:14:33.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bdffb640 1 monclient(hunting): continuing hunt 2024-03-28T16:14:33.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bdffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T16:14:33.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bdffb640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe6a0007390 legacy=0x7fe6a0009850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T16:14:33.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bdffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe6a00036e0 legacy=0x7fe6a0005b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T16:14:33.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bffff640 1 -- reap_dead start 2024-03-28T16:14:33.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bdffb640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-03-28T16:14:33.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bdffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe6a000c280 0x7fe6a0009850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:33.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bdffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe6a000c280 addr v2:172.21.15.83:3300/0 2024-03-28T16:14:33.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bdffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6a000c660 addr v1:172.21.15.83:6789/0 2024-03-28T16:14:33.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bdffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe6a0009f80 0x7fe6a000a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:33.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bdffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe6a0009f80 addr v2:172.21.15.7:3300/0 2024-03-28T16:14:33.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bdffb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a000d2c0 con 0x7fe6a000c660 2024-03-28T16:14:33.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bdffb640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:33.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bdffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6c00aa480 con 0x7fe6a0009f80 2024-03-28T16:14:33.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bdffb640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:33.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bdffb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6c0071d00 con 0x7fe6a000c280 2024-03-28T16:14:33.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bdffb640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:33.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe6a0009f80 msgr2=0x7fe6a000a430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T16:14:33.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe6a0009f80 0x7fe6a000a430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T16:14:33.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bf7fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe6a000c280 msgr2=0x7fe6a0009850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T16:14:33.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6b6ffd640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe6a000c660 legacy=0x7fe6a000ca80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T16:14:33.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.452+0000 7fe6bf7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe6a000c280 0x7fe6a0009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T16:14:33.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.653+0000 7fe6bffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe6a0009f80 msgr2=0x7fe6a000a430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T16:14:33.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.653+0000 7fe6bffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe6a0009f80 0x7fe6a000a430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T16:14:33.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.653+0000 7fe6bf7fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe6a000c280 msgr2=0x7fe6a0009850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T16:14:33.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.653+0000 7fe6b6ffd640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe6a000c660 legacy=0x7fe6a000ca80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T16:14:33.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:33.653+0000 7fe6bf7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe6a000c280 0x7fe6a0009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T16:14:34.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:34.054+0000 7fe6bffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe6a0009f80 msgr2=0x7fe6a000a430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T16:14:34.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:34.054+0000 7fe6bffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe6a0009f80 0x7fe6a000a430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T16:14:34.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:34.054+0000 7fe6bf7fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe6a000c280 msgr2=0x7fe6a0009850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T16:14:34.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:34.054+0000 7fe6b6ffd640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe6a000c660 legacy=0x7fe6a000ca80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T16:14:34.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:34.054+0000 7fe6bf7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe6a000c280 0x7fe6a0009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T16:14:34.298 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:00:06.854 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:14:34.605 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:00:07.161 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:14:34.783 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:00:07.356 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:14:34.784 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:00:07.357 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:14:34.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:34.855+0000 7fe6bffff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe6a0009f80 msgr2=0x7fe6a000a430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T16:14:34.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:34.855+0000 7fe6bffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe6a0009f80 0x7fe6a000a430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T16:14:34.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:34.855+0000 7fe6bf7fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe6a000c280 msgr2=0x7fe6a0009850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T16:14:34.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:34.855+0000 7fe6b6ffd640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe6a000c660 legacy=0x7fe6a000ca80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T16:14:34.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:34.855+0000 7fe6bf7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe6a000c280 0x7fe6a0009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T16:14:35.110 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:00:07.682 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:14:35.110 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:00:07.683 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:14:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.452+0000 7fe6bdffb640 10 monclient(hunting): tick 2024-03-28T16:14:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.452+0000 7fe6bdffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T16:14:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.452+0000 7fe6bdffb640 1 monclient(hunting): continuing hunt 2024-03-28T16:14:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.452+0000 7fe6bdffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T16:14:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.452+0000 7fe6bdffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe6a000c280 msgr2=0x7fe6a0009850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T16:14:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.452+0000 7fe6bdffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe6a000c280 0x7fe6a0009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.452+0000 7fe6bdffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe6a0009f80 msgr2=0x7fe6a000a430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T16:14:36.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.452+0000 7fe6bdffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe6a0009f80 0x7fe6a000a430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.452+0000 7fe6bdffb640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe6a000c660 legacy=0x7fe6a000ca80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T16:14:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.452+0000 7fe6bdffb640 10 monclient: _add_conns ranks=[2,0] 2024-03-28T16:14:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.452+0000 7fe6bdffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 0x7fe6a0005eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.452+0000 7fe6bdffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe6a0005ad0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.452+0000 7fe6bdffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe6a00063f0 addr v1:172.21.15.7:6789/0 2024-03-28T16:14:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.452+0000 7fe6bdffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c00aa660 con 0x7fe6a00063f0 2024-03-28T16:14:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.452+0000 7fe6bdffb640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.452+0000 7fe6bdffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c00aa480 con 0x7fe6a0005ad0 2024-03-28T16:14:36.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.452+0000 7fe6bdffb640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:36.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.657+0000 7fe6bf7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 0x7fe6a0005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:36.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.673+0000 7fe6b6ffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe6a00063f0 0x7fe6a0006800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55552/0 (socket says 172.21.15.7:55552) 2024-03-28T16:14:36.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.674+0000 7fe6b6ffd640 1 -- 172.21.15.7:0/3675351412 learned_addr learned my addr 172.21.15.7:0/3675351412 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-28T16:14:36.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.741+0000 7fe6bf7fe640 10 monclient(hunting): get_auth_request con 0x7fe6a0005ad0 auth_method 0 2024-03-28T16:14:36.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.741+0000 7fe6bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:36.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.741+0000 7fe6bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:36.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.741+0000 7fe6bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:36.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.823+0000 7fe6bf7fe640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 msgr2=0x7fe6a0005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-28T16:14:36.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.823+0000 7fe6bf7fe640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 msgr2=0x7fe6a0005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T16:14:36.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.823+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 0x7fe6a0005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T16:14:36.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.823+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 0x7fe6a0005eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T16:14:36.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.828+0000 7fe6b6ffd640 1 -- 172.21.15.7:0/3675351412 >> v1:172.21.15.7:6789/0 conn(0x7fe6a00063f0 legacy=0x7fe6a0006800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-28T16:14:36.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.828+0000 7fe6b6ffd640 1 -- 172.21.15.7:0/3675351412 >> v1:172.21.15.7:6789/0 conn(0x7fe6a00063f0 legacy=0x7fe6a0006800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T16:14:36.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.828+0000 7fe6b6ffd640 1 --1- 172.21.15.7:0/3675351412 >> v1:172.21.15.7:6789/0 conn(0x7fe6a00063f0 0x7fe6a0006800 :-1 s=OPENED pgs=1 cs=1 l=1).handle_message read tag failed 2024-03-28T16:14:36.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.828+0000 7fe6b6ffd640 1 --1- 172.21.15.7:0/3675351412 >> v1:172.21.15.7:6789/0 conn(0x7fe6a00063f0 0x7fe6a0006800 :-1 s=OPENED pgs=1 cs=1 l=1).fault on lossy channel, failing 2024-03-28T16:14:36.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:36.828+0000 7fe6beffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.7:6789/0 2024-03-28T16:14:37.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:37.312+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 0x7fe6a0005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:37.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:37.312+0000 7fe6bf7fe640 10 monclient(hunting): get_auth_request con 0x7fe6a0005ad0 auth_method 0 2024-03-28T16:14:37.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:37.312+0000 7fe6bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:37.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:37.312+0000 7fe6bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:37.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:37.312+0000 7fe6bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:14:37.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:37.318+0000 7fe6bf7fe640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 msgr2=0x7fe6a0005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-28T16:14:37.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:37.318+0000 7fe6bf7fe640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 msgr2=0x7fe6a0005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T16:14:37.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:37.318+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 0x7fe6a0005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T16:14:37.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:37.319+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 0x7fe6a0005eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T16:14:37.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:37.809+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 0x7fe6a0005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:37.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:37.810+0000 7fe6bf7fe640 10 monclient(hunting): get_auth_request con 0x7fe6a0005ad0 auth_method 0 2024-03-28T16:14:37.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:37.810+0000 7fe6bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:37.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:37.810+0000 7fe6bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:37.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:37.810+0000 7fe6bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:14:37.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:37.813+0000 7fe6bf7fe640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 msgr2=0x7fe6a0005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-28T16:14:37.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:37.813+0000 7fe6bf7fe640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 msgr2=0x7fe6a0005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T16:14:37.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:37.813+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 0x7fe6a0005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T16:14:37.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:37.813+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 0x7fe6a0005eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T16:14:38.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:38.614+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 0x7fe6a0005eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:38.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:38.615+0000 7fe6bf7fe640 10 monclient(hunting): get_auth_request con 0x7fe6a0005ad0 auth_method 0 2024-03-28T16:14:38.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:38.615+0000 7fe6bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:38.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:38.615+0000 7fe6bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:38.615 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:38.615+0000 7fe6bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:14:38.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:38.615+0000 7fe6bf7fe640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 msgr2=0x7fe6a0005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-28T16:14:38.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:38.615+0000 7fe6bf7fe640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 msgr2=0x7fe6a0005eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T16:14:38.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:38.615+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 0x7fe6a0005eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T16:14:38.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:38.615+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 0x7fe6a0005eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T16:14:39.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bdffb640 10 monclient(hunting): tick 2024-03-28T16:14:39.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bdffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T16:14:39.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bdffb640 1 monclient(hunting): continuing hunt 2024-03-28T16:14:39.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bdffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T16:14:39.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bdffb640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 msgr2=0x7fe6a0005eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T16:14:39.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bdffb640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a0005ad0 0x7fe6a0005eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:39.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bdffb640 1 -- 172.21.15.7:0/3675351412 >> v1:172.21.15.7:6789/0 conn(0x7fe6a00063f0 legacy=0x7fe6a0006800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-28T16:14:39.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bffff640 1 -- 172.21.15.7:0/3675351412 reap_dead start 2024-03-28T16:14:39.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bdffb640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-03-28T16:14:39.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bdffb640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 0x7fe6a0009850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:39.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bdffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe6a000a430 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:39.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bdffb640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 0x7fe6a000b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:39.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bdffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe6a000ca80 addr v2:172.21.15.7:3300/0 2024-03-28T16:14:39.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bdffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe6a000b5f0 addr v1:172.21.15.7:6789/0 2024-03-28T16:14:39.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bdffb640 1 -- 172.21.15.7:0/3675351412 --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a000d2c0 con 0x7fe6a000b5f0 2024-03-28T16:14:39.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bdffb640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:39.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bdffb640 1 -- 172.21.15.7:0/3675351412 --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6c00aa480 con 0x7fe6a000ca80 2024-03-28T16:14:39.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bdffb640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:39.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bdffb640 1 -- 172.21.15.7:0/3675351412 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c0071d00 con 0x7fe6a000a430 2024-03-28T16:14:39.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bdffb640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:39.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 0x7fe6a0009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:39.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bf7fe640 10 monclient(hunting): get_auth_request con 0x7fe6a000a430 auth_method 0 2024-03-28T16:14:39.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:39.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:39.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.453+0000 7fe6bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:39.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.454+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 0x7fe6a000b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:39.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.454+0000 7fe6b6ffd640 10 monclient(hunting): get_auth_request con 0x7fe6a000ca80 auth_method 0 2024-03-28T16:14:39.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.454+0000 7fe6b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:39.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.454+0000 7fe6b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:39.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.454+0000 7fe6b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:39.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.454+0000 7fe6bf7fe640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 msgr2=0x7fe6a0009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-28T16:14:39.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.454+0000 7fe6bf7fe640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 msgr2=0x7fe6a0009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T16:14:39.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.454+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 0x7fe6a0009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T16:14:39.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.454+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 0x7fe6a0009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T16:14:39.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.455+0000 7fe6b6ffd640 1 -- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 msgr2=0x7fe6a000b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-28T16:14:39.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.455+0000 7fe6b6ffd640 1 -- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 msgr2=0x7fe6a000b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T16:14:39.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.455+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 0x7fe6a000b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T16:14:39.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.455+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 0x7fe6a000b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T16:14:39.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.655+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 0x7fe6a0009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:39.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.656+0000 7fe6bf7fe640 10 monclient(hunting): get_auth_request con 0x7fe6a000a430 auth_method 0 2024-03-28T16:14:39.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.656+0000 7fe6bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:39.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.656+0000 7fe6bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:39.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.656+0000 7fe6bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:14:39.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.656+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 0x7fe6a000b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:39.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.656+0000 7fe6b6ffd640 10 monclient(hunting): get_auth_request con 0x7fe6a000ca80 auth_method 0 2024-03-28T16:14:39.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.656+0000 7fe6b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:39.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.656+0000 7fe6b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:39.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.656+0000 7fe6b6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:14:39.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.656+0000 7fe6bf7fe640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 msgr2=0x7fe6a0009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-28T16:14:39.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.656+0000 7fe6bf7fe640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 msgr2=0x7fe6a0009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T16:14:39.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.656+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 0x7fe6a0009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T16:14:39.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.656+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 0x7fe6a0009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T16:14:39.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.657+0000 7fe6b6ffd640 1 -- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 msgr2=0x7fe6a000b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-28T16:14:39.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.657+0000 7fe6b6ffd640 1 -- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 msgr2=0x7fe6a000b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T16:14:39.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.657+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 0x7fe6a000b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T16:14:39.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:39.657+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 0x7fe6a000b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T16:14:40.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.057+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 0x7fe6a0009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:40.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.058+0000 7fe6bf7fe640 10 monclient(hunting): get_auth_request con 0x7fe6a000a430 auth_method 0 2024-03-28T16:14:40.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.058+0000 7fe6bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:40.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.058+0000 7fe6bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:40.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.058+0000 7fe6bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:14:40.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.058+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 0x7fe6a000b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:40.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.058+0000 7fe6bf7fe640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 msgr2=0x7fe6a0009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-28T16:14:40.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.058+0000 7fe6bf7fe640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 msgr2=0x7fe6a0009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T16:14:40.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.058+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 0x7fe6a0009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T16:14:40.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.058+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 0x7fe6a0009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T16:14:40.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.058+0000 7fe6b6ffd640 10 monclient(hunting): get_auth_request con 0x7fe6a000ca80 auth_method 0 2024-03-28T16:14:40.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.058+0000 7fe6b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:40.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.058+0000 7fe6b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:40.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.058+0000 7fe6b6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:14:40.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.059+0000 7fe6b6ffd640 1 -- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 msgr2=0x7fe6a000b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-28T16:14:40.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.059+0000 7fe6b6ffd640 1 -- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 msgr2=0x7fe6a000b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T16:14:40.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.059+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 0x7fe6a000b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T16:14:40.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.059+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 0x7fe6a000b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T16:14:40.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.860+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 0x7fe6a0009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:40.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.860+0000 7fe6bf7fe640 10 monclient(hunting): get_auth_request con 0x7fe6a000a430 auth_method 0 2024-03-28T16:14:40.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.860+0000 7fe6bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:40.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.860+0000 7fe6bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:40.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.860+0000 7fe6bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:14:40.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.861+0000 7fe6bf7fe640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 msgr2=0x7fe6a0009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-28T16:14:40.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.861+0000 7fe6bf7fe640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 msgr2=0x7fe6a0009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T16:14:40.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.861+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 0x7fe6a0009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T16:14:40.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.861+0000 7fe6bf7fe640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 0x7fe6a0009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T16:14:40.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.861+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 0x7fe6a000b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:40.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.861+0000 7fe6b6ffd640 10 monclient(hunting): get_auth_request con 0x7fe6a000ca80 auth_method 0 2024-03-28T16:14:40.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.861+0000 7fe6b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:40.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.861+0000 7fe6b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:40.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.861+0000 7fe6b6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:14:40.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.862+0000 7fe6b6ffd640 1 -- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 msgr2=0x7fe6a000b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-28T16:14:40.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.862+0000 7fe6b6ffd640 1 -- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 msgr2=0x7fe6a000b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T16:14:40.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.862+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 0x7fe6a000b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T16:14:40.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:40.862+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 0x7fe6a000b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T16:14:42.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.453+0000 7fe6bdffb640 10 monclient(hunting): tick 2024-03-28T16:14:42.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.453+0000 7fe6bdffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T16:14:42.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.453+0000 7fe6bdffb640 1 monclient(hunting): continuing hunt 2024-03-28T16:14:42.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.453+0000 7fe6bdffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T16:14:42.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.453+0000 7fe6bdffb640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 msgr2=0x7fe6a0009850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T16:14:42.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.453+0000 7fe6bdffb640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000a430 0x7fe6a0009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:42.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.453+0000 7fe6bdffb640 1 -- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 msgr2=0x7fe6a000b0b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T16:14:42.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.453+0000 7fe6bdffb640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6a000ca80 0x7fe6a000b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:42.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.453+0000 7fe6bdffb640 1 -- 172.21.15.7:0/3675351412 >> v1:172.21.15.7:6789/0 conn(0x7fe6a000b5f0 legacy=0x7fe6a0005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:42.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.453+0000 7fe6bdffb640 10 monclient: _add_conns ranks=[1,4] 2024-03-28T16:14:42.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.453+0000 7fe6bdffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6a000baa0 addr v1:172.21.15.83:6789/0 2024-03-28T16:14:42.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.453+0000 7fe6bdffb640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 0x7fe6a00150e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:42.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.453+0000 7fe6bdffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe6a0012cf0 addr v2:172.21.15.83:3300/0 2024-03-28T16:14:42.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.453+0000 7fe6bdffb640 1 -- 172.21.15.7:0/3675351412 --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a000c0c0 con 0x7fe6a000baa0 2024-03-28T16:14:42.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.453+0000 7fe6bdffb640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:42.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.453+0000 7fe6bdffb640 1 -- 172.21.15.7:0/3675351412 --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6c00aa480 con 0x7fe6a0012cf0 2024-03-28T16:14:42.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.453+0000 7fe6bdffb640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:42.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.454+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 0x7fe6a00150e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:42.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.454+0000 7fe6b6ffd640 10 monclient(hunting): get_auth_request con 0x7fe6a0012cf0 auth_method 0 2024-03-28T16:14:42.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.454+0000 7fe6b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:42.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.454+0000 7fe6b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:42.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.454+0000 7fe6b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:42.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.456+0000 7fe6b6ffd640 1 -- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 msgr2=0x7fe6a00150e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-28T16:14:42.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.456+0000 7fe6b6ffd640 1 -- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 msgr2=0x7fe6a00150e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T16:14:42.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.456+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 0x7fe6a00150e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T16:14:42.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.456+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 0x7fe6a00150e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T16:14:42.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.658+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 0x7fe6a00150e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:42.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.658+0000 7fe6b6ffd640 10 monclient(hunting): get_auth_request con 0x7fe6a0012cf0 auth_method 0 2024-03-28T16:14:42.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.658+0000 7fe6b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:42.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.658+0000 7fe6b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:42.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.658+0000 7fe6b6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:14:42.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.658+0000 7fe6b6ffd640 1 -- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 msgr2=0x7fe6a00150e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-28T16:14:42.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.658+0000 7fe6b6ffd640 1 -- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 msgr2=0x7fe6a00150e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T16:14:42.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.658+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 0x7fe6a00150e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T16:14:42.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:42.658+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 0x7fe6a00150e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T16:14:43.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:43.060+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 0x7fe6a00150e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:43.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:43.060+0000 7fe6b6ffd640 10 monclient(hunting): get_auth_request con 0x7fe6a0012cf0 auth_method 0 2024-03-28T16:14:43.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:43.060+0000 7fe6b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:43.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:43.060+0000 7fe6b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:43.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:43.060+0000 7fe6b6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:14:43.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:43.061+0000 7fe6b6ffd640 1 -- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 msgr2=0x7fe6a00150e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-28T16:14:43.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:43.061+0000 7fe6b6ffd640 1 -- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 msgr2=0x7fe6a00150e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T16:14:43.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:43.061+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 0x7fe6a00150e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T16:14:43.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:43.061+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 0x7fe6a00150e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T16:14:43.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:43.862+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 0x7fe6a00150e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:43.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:43.862+0000 7fe6b6ffd640 10 monclient(hunting): get_auth_request con 0x7fe6a0012cf0 auth_method 0 2024-03-28T16:14:43.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:43.862+0000 7fe6b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:43.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:43.862+0000 7fe6b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:43.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:43.862+0000 7fe6b6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:14:43.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:43.864+0000 7fe6b6ffd640 1 -- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 msgr2=0x7fe6a00150e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-28T16:14:43.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:43.864+0000 7fe6b6ffd640 1 -- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 msgr2=0x7fe6a00150e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T16:14:43.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:43.864+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 0x7fe6a00150e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T16:14:43.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:43.864+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 0x7fe6a00150e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T16:14:45.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.452+0000 7fe6bdffb640 10 monclient(hunting): tick 2024-03-28T16:14:45.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.452+0000 7fe6bdffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T16:14:45.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.452+0000 7fe6bdffb640 1 monclient(hunting): continuing hunt 2024-03-28T16:14:45.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.452+0000 7fe6bdffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T16:14:45.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.452+0000 7fe6bdffb640 1 -- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 msgr2=0x7fe6a00150e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T16:14:45.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.452+0000 7fe6bdffb640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a0012cf0 0x7fe6a00150e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:45.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.452+0000 7fe6bdffb640 1 -- 172.21.15.7:0/3675351412 >> v1:172.21.15.83:6789/0 conn(0x7fe6a000baa0 legacy=0x7fe6a00115d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:45.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.452+0000 7fe6bdffb640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-03-28T16:14:45.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.452+0000 7fe6bffff640 1 -- 172.21.15.7:0/3675351412 reap_dead start 2024-03-28T16:14:45.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.452+0000 7fe6bdffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6a00156a0 addr v1:172.21.15.83:6789/0 2024-03-28T16:14:45.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.452+0000 7fe6bdffb640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a000a430 0x7fe6a0009850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:45.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.452+0000 7fe6bdffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe6a000a430 addr v2:172.21.15.83:3300/0 2024-03-28T16:14:45.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.452+0000 7fe6bdffb640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000ca80 0x7fe6a000b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:45.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.452+0000 7fe6bdffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe6a000ca80 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:45.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.452+0000 7fe6bdffb640 1 -- 172.21.15.7:0/3675351412 --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a000d070 con 0x7fe6a00156a0 2024-03-28T16:14:45.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.452+0000 7fe6bdffb640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:45.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.452+0000 7fe6bdffb640 1 -- 172.21.15.7:0/3675351412 --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6c00aa480 con 0x7fe6a000a430 2024-03-28T16:14:45.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.452+0000 7fe6bdffb640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:45.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.452+0000 7fe6bdffb640 1 -- 172.21.15.7:0/3675351412 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c0071d00 con 0x7fe6a000ca80 2024-03-28T16:14:45.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.452+0000 7fe6bdffb640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.486+0000 7fe6bffff640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000ca80 0x7fe6a000b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.487+0000 7fe6bffff640 10 monclient(hunting): get_auth_request con 0x7fe6a000ca80 auth_method 0 2024-03-28T16:14:45.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.487+0000 7fe6bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:45.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.487+0000 7fe6bffff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:45.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.487+0000 7fe6bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:45.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.488+0000 7fe6bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:45.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.488+0000 7fe6bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:45.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.488+0000 7fe6bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:45.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.491+0000 7fe6bffff640 10 monclient(hunting): handle_auth_done global_id 4106 payload 274 2024-03-28T16:14:45.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.491+0000 7fe6bffff640 1 -- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a000a430 msgr2=0x7fe6a0009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:14:45.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.491+0000 7fe6bffff640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.83:3300/0 conn(0x7fe6a000a430 0x7fe6a0009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:45.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.491+0000 7fe6bffff640 1 -- 172.21.15.7:0/3675351412 >> v1:172.21.15.83:6789/0 conn(0x7fe6a00156a0 legacy=0x7fe6a0017a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:14:45.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.491+0000 7fe6bffff640 10 monclient: _finish_hunting 0 2024-03-28T16:14:45.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.491+0000 7fe6bffff640 1 monclient: found mon.noname-b 2024-03-28T16:14:45.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.491+0000 7fe6bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:45.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.491+0000 7fe6bffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-03-28T16:14:45.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.491+0000 7fe6bffff640 1 -- 172.21.15.7:0/3675351412 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a000b5f0 con 0x7fe6a000ca80 2024-03-28T16:14:45.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.509+0000 7fe6bffff640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000ca80 0x7fe6a000b0b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe6c00b1c70 tx=0x7fe6b802b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=847016d624186f40 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.511+0000 7fe6bffff640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000ca80 msgr2=0x7fe6a000b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-28T16:14:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.511+0000 7fe6bffff640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000ca80 msgr2=0x7fe6a000b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T16:14:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.511+0000 7fe6bffff640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000ca80 0x7fe6a000b0b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe6c00b1c70 tx=0x7fe6b802b770 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T16:14:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.511+0000 7fe6bffff640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000ca80 0x7fe6a000b0b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe6c00b1c70 tx=0x7fe6b802b770 comp rx=0 tx=0).stop 2024-03-28T16:14:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.511+0000 7fe6beffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.511+0000 7fe6beffd640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.511+0000 7fe6beffd640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000ca80 msgr2=0x7fe6a000b0b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-28T16:14:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.511+0000 7fe6beffd640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6a000ca80 0x7fe6a000b0b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.511+0000 7fe6beffd640 10 monclient: _add_conns ranks=[0,3] 2024-03-28T16:14:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.511+0000 7fe6beffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe6ac003500 addr v1:172.21.15.7:6789/0 2024-03-28T16:14:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.511+0000 7fe6beffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6ac007490 0x7fe6ac009950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.511+0000 7fe6beffd640 10 monclient(hunting): picked mon.noname-a con 0x7fe6ac007490 addr v2:172.21.15.7:3300/0 2024-03-28T16:14:45.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.511+0000 7fe6beffd640 1 -- 172.21.15.7:0/3675351412 --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6ac00a220 con 0x7fe6ac003500 2024-03-28T16:14:45.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.511+0000 7fe6beffd640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:45.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.511+0000 7fe6beffd640 1 -- 172.21.15.7:0/3675351412 --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6ac00b3e0 con 0x7fe6ac007490 2024-03-28T16:14:45.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.511+0000 7fe6beffd640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:45.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.511+0000 7fe6beffd640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:45.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.537+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> v2:172.21.15.7:3300/0 conn(0x7fe6ac007490 0x7fe6ac009950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:45.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.574+0000 7fe6b6ffd640 10 monclient(hunting): get_auth_request con 0x7fe6ac007490 auth_method 0 2024-03-28T16:14:45.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.574+0000 7fe6b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:45.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.574+0000 7fe6b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:45.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.574+0000 7fe6b6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:14:45.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.576+0000 7fe6b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:45.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.576+0000 7fe6b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:45.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.576+0000 7fe6b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-28T16:14:45.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.588+0000 7fe6b6ffd640 10 monclient(hunting): handle_auth_done global_id 4106 payload 293 2024-03-28T16:14:45.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.588+0000 7fe6b6ffd640 1 -- 172.21.15.7:0/3675351412 >> v1:172.21.15.7:6789/0 conn(0x7fe6ac003500 legacy=0x7fe6ac005980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:14:45.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.588+0000 7fe6b6ffd640 10 monclient: _finish_hunting 0 2024-03-28T16:14:45.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.588+0000 7fe6b6ffd640 1 monclient: found mon.noname-a 2024-03-28T16:14:45.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.588+0000 7fe6b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:45.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.588+0000 7fe6b6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:14:45.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.588+0000 7fe6b6ffd640 1 -- 172.21.15.7:0/3675351412 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6ac00b610 con 0x7fe6ac007490 2024-03-28T16:14:45.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.594+0000 7fe6b6ffd640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6ac007490 0x7fe6ac009950 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe6a80095b0 tx=0x7fe6b80073f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef46268610017c4e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:45.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.601+0000 7fe6beffd640 1 -- 172.21.15.7:0/3675351412 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a800e070 con 0x7fe6ac007490 2024-03-28T16:14:45.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.601+0000 7fe6beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:45.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.601+0000 7fe6beffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:14:45.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.601+0000 7fe6beffd640 10 monclient: dump: 2024-03-28T16:14:45.603 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:14:45.603 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:45.603 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:45.603 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:45.603 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:45.603 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:45.603 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:45.603 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:45.604 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:45.604 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:14:45.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.601+0000 7fe6beffd640 10 monclient: _finish_auth 0 2024-03-28T16:14:45.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.601+0000 7fe6beffd640 10 monclient: _check_auth_tickets 2024-03-28T16:14:45.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.601+0000 7fe6beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:45.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.601+0000 7fe6beffd640 1 -- 172.21.15.7:0/3675351412 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a8005790 con 0x7fe6ac007490 2024-03-28T16:14:45.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.601+0000 7fe6beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:45.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.601+0000 7fe6beffd640 1 -- 172.21.15.7:0/3675351412 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a801a440 con 0x7fe6ac007490 2024-03-28T16:14:45.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.601+0000 7fe6c5c56640 5 monclient: authenticate success, global_id 4106 2024-03-28T16:14:45.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.601+0000 7fe6c5c56640 10 monclient: get_monmap_and_config success 2024-03-28T16:14:45.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.601+0000 7fe6c5c56640 4 set_mon_vals no callback set 2024-03-28T16:14:45.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.601+0000 7fe6beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:45.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.601+0000 7fe6beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:14:45.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.601+0000 7fe6beffd640 10 monclient: dump: 2024-03-28T16:14:45.604 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:14:45.604 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:45.605 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:45.605 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:45.605 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:45.605 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:45.605 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:45.605 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:45.605 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:45.605 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:14:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.602+0000 7fe6c5c56640 10 monclient: shutdown 2024-03-28T16:14:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.602+0000 7fe6c5c56640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.602+0000 7fe6c5c56640 1 -- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6ac007490 msgr2=0x7fe6ac009950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.602+0000 7fe6c5c56640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6ac007490 0x7fe6ac009950 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe6a80095b0 tx=0x7fe6b80073f0 comp rx=0 tx=0).stop 2024-03-28T16:14:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.602+0000 7fe6bffff640 1 -- 172.21.15.7:0/3675351412 reap_dead start 2024-03-28T16:14:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.602+0000 7fe6c5c56640 1 -- 172.21.15.7:0/3675351412 shutdown_connections 2024-03-28T16:14:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.602+0000 7fe6c5c56640 1 --2- 172.21.15.7:0/3675351412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6ac007490 0x7fe6ac009950 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.602+0000 7fe6c5c56640 1 -- 172.21.15.7:0/3675351412 >> 172.21.15.7:0/3675351412 conn(0x7fe6c00b3db0 msgr2=0x7fe6c00b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.602+0000 7fe6c5c56640 1 -- 172.21.15.7:0/3675351412 shutdown_connections 2024-03-28T16:14:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.602+0000 7fe6c5c56640 1 -- 172.21.15.7:0/3675351412 wait complete. 2024-03-28T16:14:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.603+0000 7fe6c5c56640 10 monclient: build_initial_monmap 2024-03-28T16:14:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.603+0000 7fe6c5c56640 10 monclient: monmap: 2024-03-28T16:14:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:14:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:14:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:14:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:14:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:14:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:14:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:14:45.609 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:14:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.603+0000 7fe6c5c56640 1 librados: starting msgr at 2024-03-28T16:14:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.603+0000 7fe6c5c56640 1 librados: starting objecter 2024-03-28T16:14:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.603+0000 7fe6c5c56640 1 Processor -- start 2024-03-28T16:14:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.604+0000 7fe6c5c56640 1 -- start start 2024-03-28T16:14:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.604+0000 7fe6c5c56640 1 librados: setting wanted keys 2024-03-28T16:14:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.604+0000 7fe6c5c56640 1 librados: calling monclient init 2024-03-28T16:14:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.604+0000 7fe6c5c56640 10 monclient: init 2024-03-28T16:14:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.604+0000 7fe6c5c56640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.604+0000 7fe6c5c56640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-28T16:14:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.604+0000 7fe6c5c56640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6ac007490 0x7fe6c00aacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.604+0000 7fe6c5c56640 10 monclient(hunting): picked mon.noname-c con 0x7fe6ac007490 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.604+0000 7fe6c5c56640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6c00ab2e0 0x7fe6c01d9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.604+0000 7fe6c5c56640 10 monclient(hunting): picked mon.noname-b con 0x7fe6c00ab2e0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:14:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.604+0000 7fe6c5c56640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6c01da100 0x7fe6c01d6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.604+0000 7fe6c5c56640 10 monclient(hunting): picked mon.noname-a con 0x7fe6c01da100 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:14:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.604+0000 7fe6c5c56640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.604+0000 7fe6c5c56640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c0145840 con 0x7fe6c01da100 2024-03-28T16:14:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.604+0000 7fe6c5c56640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.604+0000 7fe6c5c56640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c00ac3f0 con 0x7fe6c00ab2e0 2024-03-28T16:14:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.604+0000 7fe6c5c56640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.604+0000 7fe6c5c56640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c0144310 con 0x7fe6ac007490 2024-03-28T16:14:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.604+0000 7fe6c5c56640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:45.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.604+0000 7fe6c5c56640 10 monclient(hunting): authenticate will time out at 1352.746567s 2024-03-28T16:14:45.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.617+0000 7fe6bf7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6ac007490 0x7fe6c00aacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:45.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.621+0000 7fe6bffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6c01da100 0x7fe6c01d6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:45.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.621+0000 7fe6bffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6c01da100 0x7fe6c01d6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49882/0 (socket says 172.21.15.7:49882) 2024-03-28T16:14:45.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.621+0000 7fe6bffff640 1 -- 172.21.15.7:0/1764821447 learned_addr learned my addr 172.21.15.7:0/1764821447 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:14:45.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.621+0000 7fe6bffff640 10 monclient(hunting): get_auth_request con 0x7fe6c01da100 auth_method 0 2024-03-28T16:14:45.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.621+0000 7fe6bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:45.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.621+0000 7fe6bffff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:45.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.621+0000 7fe6bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:45.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.622+0000 7fe6bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:45.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.622+0000 7fe6bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:45.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.622+0000 7fe6bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:45.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.633+0000 7fe6bffff640 10 monclient(hunting): handle_auth_done global_id 4113 payload 824 2024-03-28T16:14:45.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.633+0000 7fe6bffff640 1 -- 172.21.15.7:0/1764821447 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6ac007490 msgr2=0x7fe6c00aacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:45.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.633+0000 7fe6bffff640 1 --2- 172.21.15.7:0/1764821447 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6ac007490 0x7fe6c00aacb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:45.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.633+0000 7fe6bffff640 1 -- 172.21.15.7:0/1764821447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6c00ab2e0 msgr2=0x7fe6c01d9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:45.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.633+0000 7fe6bffff640 1 --2- 172.21.15.7:0/1764821447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6c00ab2e0 0x7fe6c01d9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:45.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.633+0000 7fe6bffff640 10 monclient: _finish_hunting 0 2024-03-28T16:14:45.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.633+0000 7fe6bffff640 1 monclient: found mon.noname-a 2024-03-28T16:14:45.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.633+0000 7fe6bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:45.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.633+0000 7fe6bffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:14:45.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.633+0000 7fe6bffff640 1 -- 172.21.15.7:0/1764821447 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c0145380 con 0x7fe6c01da100 2024-03-28T16:14:45.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.636+0000 7fe6bffff640 1 --2- 172.21.15.7:0/1764821447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6c01da100 0x7fe6c01d6330 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8003ed0 tx=0x7fe6b8005f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:45.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.636+0000 7fe6bcff9640 10 client.?.objecter ms_handle_connect 0x7fe6c01da100 2024-03-28T16:14:45.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.636+0000 7fe6bcff9640 10 client.?.objecter resend_mon_ops 2024-03-28T16:14:45.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.639+0000 7fe6bcff9640 1 -- 172.21.15.7:0/1764821447 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b802f5c0 con 0x7fe6c01da100 2024-03-28T16:14:45.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:45.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6bcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:14:45.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6bcff9640 10 monclient: dump: 2024-03-28T16:14:45.641 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:14:45.641 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:45.642 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:45.642 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:45.642 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:45.642 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:45.642 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:45.642 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:45.642 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:45.642 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:14:45.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6bcff9640 10 monclient: _finish_auth 0 2024-03-28T16:14:45.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6bcff9640 10 monclient: _check_auth_tickets 2024-03-28T16:14:45.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:45.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6bcff9640 1 -- 172.21.15.7:0/1764821447 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b802f720 con 0x7fe6c01da100 2024-03-28T16:14:45.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6bcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:45.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6bcff9640 1 -- 172.21.15.7:0/1764821447 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b802fa90 con 0x7fe6c01da100 2024-03-28T16:14:45.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6c5c56640 5 monclient: authenticate success, global_id 4113 2024-03-28T16:14:45.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6c5c56640 10 monclient: _renew_subs 2024-03-28T16:14:45.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6c5c56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:14:45.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6c5c56640 1 -- 172.21.15.7:0/1764821447 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6c01da580 con 0x7fe6c01da100 2024-03-28T16:14:45.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6be7fc640 4 set_mon_vals no callback set 2024-03-28T16:14:45.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:45.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6bcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:14:45.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6bcff9640 10 monclient: dump: 2024-03-28T16:14:45.643 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:14:45.643 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:45.643 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:45.643 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:45.643 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:45.643 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:45.643 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:45.644 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:45.644 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:45.644 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:14:45.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6c5c56640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:14:45.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6c5c56640 10 monclient: _renew_subs 2024-03-28T16:14:45.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6c5c56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:14:45.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6c5c56640 1 -- 172.21.15.7:0/1764821447 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6c01d6a50 con 0x7fe6c01da100 2024-03-28T16:14:45.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.640+0000 7fe6c5c56640 1 librados: init done 2024-03-28T16:14:45.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.641+0000 7fe6c5c56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:45.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.641+0000 7fe6c5c56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:45.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.641+0000 7fe6c5c56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:14:45.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.641+0000 7fe6c5c56640 1 -- 172.21.15.7:0/1764821447 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6c01d98e0 con 0x7fe6c01da100 2024-03-28T16:14:45.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.660+0000 7fe6bcff9640 1 -- 172.21.15.7:0/1764821447 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe6b8040420 con 0x7fe6c01da100 2024-03-28T16:14:45.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.660+0000 7fe6bcff9640 1 -- 172.21.15.7:0/1764821447 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe6b80493f0 con 0x7fe6c01da100 2024-03-28T16:14:45.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.660+0000 7fe6bcff9640 10 client.4113.objecter ms_dispatch 0x7fe6c0000b90 osd_map(1..1 src has 1..1) v4 2024-03-28T16:14:45.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.660+0000 7fe6bcff9640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-28T16:14:45.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.660+0000 7fe6bcff9640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-28T16:14:45.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.660+0000 7fe6bcff9640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-28T16:14:45.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.835+0000 7fe6bcff9640 1 -- 172.21.15.7:0/1764821447 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fe6c01d98e0 con 0x7fe6c01da100 2024-03-28T16:14:45.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.835+0000 7fe6bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:45.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.835+0000 7fe6bcff9640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:14:45.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.926+0000 7fe6c5c56640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-28T16:14:45.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.926+0000 7fe6c5c56640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-28T16:14:45.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.926+0000 7fe6c5c56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:14:45.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:45.926+0000 7fe6c5c56640 1 -- 172.21.15.7:0/1764821447 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe6c0069430 con 0x7fe6c01da100 2024-03-28T16:14:46.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:46.083+0000 7fe6bcff9640 1 -- 172.21.15.7:0/1764821447 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe6b8045070 con 0x7fe6c01da100 2024-03-28T16:14:46.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:46.083+0000 7fe6bcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-28T16:14:46.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:46.083+0000 7fe6bcff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-28T16:14:46.084 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2024-03-28T16:14:46.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:46.084+0000 7fe6c5c56640 10 librados: watch_flush enter 2024-03-28T16:14:46.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:46.084+0000 7fe6c5c56640 10 librados: watch_flush exit 2024-03-28T16:14:46.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:46.084+0000 7fe6c5c56640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-28T16:14:46.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:46.084+0000 7fe6c5c56640 10 client.4113.objecter successfully canceled tick 2024-03-28T16:14:46.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:46.084+0000 7fe6c5c56640 10 monclient: shutdown 2024-03-28T16:14:46.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:46.084+0000 7fe6c5c56640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:46.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:46.084+0000 7fe6c5c56640 1 -- 172.21.15.7:0/1764821447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6c01da100 msgr2=0x7fe6c01d6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:46.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:46.084+0000 7fe6c5c56640 1 --2- 172.21.15.7:0/1764821447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6c01da100 0x7fe6c01d6330 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8003ed0 tx=0x7fe6b8005f20 comp rx=0 tx=0).stop 2024-03-28T16:14:46.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:46.084+0000 7fe6c5c56640 1 -- 172.21.15.7:0/1764821447 shutdown_connections 2024-03-28T16:14:46.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:46.084+0000 7fe6c5c56640 1 --2- 172.21.15.7:0/1764821447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6c01da100 0x7fe6c01d6330 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:46.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:46.084+0000 7fe6c5c56640 1 --2- 172.21.15.7:0/1764821447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6c00ab2e0 0x7fe6c01d9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:46.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:46.084+0000 7fe6c5c56640 1 --2- 172.21.15.7:0/1764821447 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe6ac007490 0x7fe6c00aacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:46.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:46.084+0000 7fe6c5c56640 1 -- 172.21.15.7:0/1764821447 >> 172.21.15.7:0/1764821447 conn(0x7fe6c00b3db0 msgr2=0x7fe6c013ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:46.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:46.084+0000 7fe6c5c56640 1 -- 172.21.15.7:0/1764821447 shutdown_connections 2024-03-28T16:14:46.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:46.084+0000 7fe6c5c56640 1 -- 172.21.15.7:0/1764821447 wait complete. 2024-03-28T16:14:46.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:46.085+0000 7fe6c5c56640 1 librados: shutdown 2024-03-28T16:14:46.097 INFO:tasks.ceph:check_enable_crimson: False 2024-03-28T16:14:46.097 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-28T16:14:46.098 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:14:46.098 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-28T16:14:46.125 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:14:46.125 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-28T16:14:46.182 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:46.182+0000 7f32d7241040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T16:14:46.192 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:14:46.192 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-28T16:14:46.195 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:46.194+0000 7f392f430040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T16:14:46.223 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:14:46.223 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-28T16:14:46.237 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:46.237+0000 7f32d7241040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T16:14:46.256 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:46.255+0000 7f392f430040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T16:14:46.283 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-28T16:14:46.283 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-28T16:14:46.311 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-28T16:14:46.311 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-28T16:14:46.378 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-28T16:14:46.378 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-28T16:14:46.444 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-28T16:14:46.444 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-28T16:14:46.512 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd new a19c6856-78b9-4919-97ce-6940a5af982c 0 2024-03-28T16:14:46.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.654+0000 7f068905c640 10 monclient: get_monmap_and_config 2024-03-28T16:14:46.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.654+0000 7f068905c640 10 monclient: build_initial_monmap 2024-03-28T16:14:46.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.654+0000 7f068905c640 10 monclient: monmap: 2024-03-28T16:14:46.654 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-28T16:14:46.654 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:46.654 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:14:46.654821+0000 2024-03-28T16:14:46.654 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:14:46.654821+0000 2024-03-28T16:14:46.654 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:46.654 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:46.654 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:14:46.655 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:14:46.655 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:14:46.655 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:14:46.655 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:14:46.655 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:46.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.654+0000 7f068905c640 1 Processor -- start 2024-03-28T16:14:46.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.654+0000 7f068905c640 1 -- start start 2024-03-28T16:14:46.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.654+0000 7f068905c640 10 monclient: init 2024-03-28T16:14:46.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.655+0000 7f068905c640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:46.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.655+0000 7f068905c640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-03-28T16:14:46.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.655+0000 7f068905c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0684071480 0x7f068413e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:46.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.655+0000 7f068905c640 10 monclient(hunting): picked mon.noname-b con 0x7f0684071480 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:46.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.655+0000 7f068905c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f06840b32e0 addr v1:172.21.15.83:6789/0 2024-03-28T16:14:46.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.655+0000 7f068905c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f068413ee50 0x7f0684145110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:46.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.655+0000 7f068905c640 10 monclient(hunting): picked mon.noname-c con 0x7f068413ee50 addr v2:172.21.15.83:3300/0 2024-03-28T16:14:46.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.655+0000 7f068905c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06840a8c90 con 0x7f06840b32e0 2024-03-28T16:14:46.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.655+0000 7f068905c640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:46.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.655+0000 7f068905c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0684071860 con 0x7f068413ee50 2024-03-28T16:14:46.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.655+0000 7f068905c640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:46.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.655+0000 7f068905c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0684145650 con 0x7f0684071480 2024-03-28T16:14:46.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.655+0000 7f068905c640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:46.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.655+0000 7f068905c640 10 monclient(hunting): authenticate will time out at 1353.824360s 2024-03-28T16:14:46.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.656+0000 7f068359e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f068413ee50 0x7f0684145110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:46.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.656+0000 7f0682d9d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0684071480 0x7f068413e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:46.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.656+0000 7f0682d9d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0684071480 0x7f068413e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.83:42506/0 (socket says 172.21.15.83:42506) 2024-03-28T16:14:46.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.656+0000 7f0682d9d640 1 -- 172.21.15.83:0/3834899230 learned_addr learned my addr 172.21.15.83:0/3834899230 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-28T16:14:46.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.656+0000 7f0682d9d640 10 monclient(hunting): get_auth_request con 0x7f0684071480 auth_method 0 2024-03-28T16:14:46.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.656+0000 7f0682d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:46.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.656+0000 7f0682d9d640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:46.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.656+0000 7f0682d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:46.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.656+0000 7f068359e640 10 monclient(hunting): get_auth_request con 0x7f068413ee50 auth_method 0 2024-03-28T16:14:46.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.656+0000 7f068359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:46.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.656+0000 7f068359e640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:46.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.656+0000 7f068359e640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:46.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.656+0000 7f0682d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:46.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.656+0000 7f0682d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:46.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.656+0000 7f0682d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:46.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.657+0000 7f068359e640 1 -- 172.21.15.83:0/3834899230 >> v2:172.21.15.83:3300/0 conn(0x7f068413ee50 msgr2=0x7f0684145110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-28T16:14:46.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.657+0000 7f068359e640 1 -- 172.21.15.83:0/3834899230 >> v2:172.21.15.83:3300/0 conn(0x7f068413ee50 msgr2=0x7f0684145110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-28T16:14:46.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.657+0000 7f068359e640 1 --2- 172.21.15.83:0/3834899230 >> v2:172.21.15.83:3300/0 conn(0x7f068413ee50 0x7f0684145110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T16:14:46.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.657+0000 7f068359e640 1 --2- 172.21.15.83:0/3834899230 >> v2:172.21.15.83:3300/0 conn(0x7f068413ee50 0x7f0684145110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T16:14:46.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.658+0000 7f0682d9d640 10 monclient(hunting): handle_auth_done global_id 4115 payload 274 2024-03-28T16:14:46.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.658+0000 7f0682d9d640 1 -- 172.21.15.83:0/3834899230 >> v2:172.21.15.83:3300/0 conn(0x7f068413ee50 msgr2=0x7f0684145110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T16:14:46.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.658+0000 7f0682d9d640 1 --2- 172.21.15.83:0/3834899230 >> v2:172.21.15.83:3300/0 conn(0x7f068413ee50 0x7f0684145110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:46.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.658+0000 7f0682d9d640 1 -- 172.21.15.83:0/3834899230 >> v1:172.21.15.83:6789/0 conn(0x7f06840b32e0 legacy=0x7f0684144a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:46.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.658+0000 7f0682d9d640 10 monclient: _finish_hunting 0 2024-03-28T16:14:46.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.658+0000 7f0682d9d640 1 monclient: found mon.noname-b 2024-03-28T16:14:46.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.658+0000 7f0682d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:46.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.658+0000 7f0682d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-03-28T16:14:46.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.658+0000 7f0682d9d640 1 -- 172.21.15.83:0/3834899230 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0684072140 con 0x7f0684071480 2024-03-28T16:14:46.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.660+0000 7f0682d9d640 1 --2- 172.21.15.83:0/3834899230 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0684071480 0x7f068413e910 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0674009870 tx=0x7f067402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36ded64f64155d10 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:46.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.664+0000 7f0681d9b640 1 -- 172.21.15.83:0/3834899230 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06740317c0 con 0x7f0684071480 2024-03-28T16:14:46.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.664+0000 7f0681d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:46.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.664+0000 7f0681d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:14:46.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.664+0000 7f0681d9b640 10 monclient: dump: 2024-03-28T16:14:46.665 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:46.665 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:46.665 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:46.665 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:46.665 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:46.665 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:46.665 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:46.665 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:46.665 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:46.665 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:46.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.664+0000 7f0681d9b640 10 monclient: _finish_auth 0 2024-03-28T16:14:46.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.664+0000 7f0681d9b640 10 monclient: _check_auth_tickets 2024-03-28T16:14:46.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.664+0000 7f0681d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:46.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.664+0000 7f0681d9b640 1 -- 172.21.15.83:0/3834899230 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0674031920 con 0x7f0684071480 2024-03-28T16:14:46.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.664+0000 7f068905c640 5 monclient: authenticate success, global_id 4115 2024-03-28T16:14:46.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.664+0000 7f068905c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:14:46.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.664+0000 7f0681d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:46.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.664+0000 7f068905c640 10 monclient: get_monmap_and_config success 2024-03-28T16:14:46.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.664+0000 7f0681d9b640 1 -- 172.21.15.83:0/3834899230 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0674031cb0 con 0x7f0684071480 2024-03-28T16:14:46.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.664+0000 7f0681d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:46.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.664+0000 7f068905c640 4 set_mon_vals no callback set 2024-03-28T16:14:46.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.664+0000 7f0681d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:14:46.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.664+0000 7f0681d9b640 10 monclient: dump: 2024-03-28T16:14:46.666 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:46.666 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:46.666 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:46.667 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:46.667 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:46.667 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:46.667 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:46.667 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:46.667 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:46.667 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:46.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.665+0000 7f068905c640 10 monclient: shutdown 2024-03-28T16:14:46.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.665+0000 7f068905c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:46.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.665+0000 7f068905c640 1 -- 172.21.15.83:0/3834899230 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0684071480 msgr2=0x7f068413e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:46.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.665+0000 7f068905c640 1 --2- 172.21.15.83:0/3834899230 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0684071480 0x7f068413e910 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0674009870 tx=0x7f067402b860 comp rx=0 tx=0).stop 2024-03-28T16:14:46.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.665+0000 7f068905c640 1 -- 172.21.15.83:0/3834899230 shutdown_connections 2024-03-28T16:14:46.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.665+0000 7f068905c640 1 --2- 172.21.15.83:0/3834899230 >> v2:172.21.15.83:3300/0 conn(0x7f068413ee50 0x7f0684145110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:46.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.665+0000 7f068905c640 1 --2- 172.21.15.83:0/3834899230 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0684071480 0x7f068413e910 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:46.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.665+0000 7f068905c640 1 -- 172.21.15.83:0/3834899230 >> 172.21.15.83:0/3834899230 conn(0x7f06840b70e0 msgr2=0x7f06840b74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:46.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.665+0000 7f068905c640 1 -- 172.21.15.83:0/3834899230 shutdown_connections 2024-03-28T16:14:46.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.665+0000 7f068905c640 1 -- 172.21.15.83:0/3834899230 wait complete. 2024-03-28T16:14:46.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.666+0000 7f068905c640 10 monclient: build_initial_monmap 2024-03-28T16:14:46.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.666+0000 7f068905c640 10 monclient: monmap: 2024-03-28T16:14:46.671 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-28T16:14:46.671 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:14:46.671 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-28T16:14:46.671 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-28T16:14:46.671 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:46.671 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:46.672 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:14:46.672 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:14:46.672 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:14:46.672 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:46.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.666+0000 7f068905c640 1 librados: starting msgr at 2024-03-28T16:14:46.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.666+0000 7f068905c640 1 librados: starting objecter 2024-03-28T16:14:46.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.667+0000 7f068905c640 1 Processor -- start 2024-03-28T16:14:46.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.667+0000 7f068905c640 1 -- start start 2024-03-28T16:14:46.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.667+0000 7f068905c640 1 librados: setting wanted keys 2024-03-28T16:14:46.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.667+0000 7f068905c640 1 librados: calling monclient init 2024-03-28T16:14:46.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.667+0000 7f068905c640 10 monclient: init 2024-03-28T16:14:46.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.667+0000 7f068905c640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:46.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.667+0000 7f068905c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-28T16:14:46.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.667+0000 7f068905c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0684071480 0x7f0684137a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:46.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.667+0000 7f068905c640 10 monclient(hunting): picked mon.noname-a con 0x7f0684071480 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:14:46.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.667+0000 7f068905c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06840b32e0 0x7f0684143c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:46.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.667+0000 7f068905c640 10 monclient(hunting): picked mon.noname-b con 0x7f06840b32e0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:14:46.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.667+0000 7f068905c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f068413ee50 0x7f06841da420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:46.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.667+0000 7f068905c640 10 monclient(hunting): picked mon.noname-c con 0x7f068413ee50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:46.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.667+0000 7f068905c640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:46.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.667+0000 7f068905c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06841d9f90 con 0x7f0684071480 2024-03-28T16:14:46.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.667+0000 7f068905c640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:46.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.667+0000 7f068905c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06841d9c60 con 0x7f06840b32e0 2024-03-28T16:14:46.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.667+0000 7f068905c640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:46.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.667+0000 7f068905c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06841431c0 con 0x7f068413ee50 2024-03-28T16:14:46.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.667+0000 7f068905c640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:46.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.667+0000 7f068905c640 10 monclient(hunting): authenticate will time out at 1353.836502s 2024-03-28T16:14:46.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.668+0000 7f068259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06840b32e0 0x7f0684143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:46.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.668+0000 7f068359e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f068413ee50 0x7f06841da420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:46.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.668+0000 7f0682d9d640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0684071480 0x7f0684137a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:46.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.668+0000 7f068259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06840b32e0 0x7f0684143c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35964/0 (socket says 172.21.15.83:35964) 2024-03-28T16:14:46.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.668+0000 7f068259c640 1 -- 172.21.15.83:0/530887049 learned_addr learned my addr 172.21.15.83:0/530887049 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-28T16:14:46.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.668+0000 7f068259c640 10 monclient(hunting): get_auth_request con 0x7f06840b32e0 auth_method 0 2024-03-28T16:14:46.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.668+0000 7f068259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:46.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.668+0000 7f068259c640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:46.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.668+0000 7f068259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:46.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.668+0000 7f068359e640 10 monclient(hunting): get_auth_request con 0x7f068413ee50 auth_method 0 2024-03-28T16:14:46.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.668+0000 7f068359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:46.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.668+0000 7f068359e640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:46.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.668+0000 7f068359e640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:46.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.668+0000 7f0682d9d640 10 monclient(hunting): get_auth_request con 0x7f0684071480 auth_method 0 2024-03-28T16:14:46.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.668+0000 7f0682d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:46.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.668+0000 7f0682d9d640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:46.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.668+0000 7f0682d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:46.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.669+0000 7f068259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:46.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.669+0000 7f068259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:46.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.669+0000 7f068259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:46.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.669+0000 7f068359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:46.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.669+0000 7f068359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:46.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.669+0000 7f068359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:46.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.669+0000 7f0682d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:46.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.669+0000 7f0682d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:46.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.669+0000 7f0682d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:46.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.672+0000 7f068259c640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2024-03-28T16:14:46.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.672+0000 7f068259c640 1 -- 172.21.15.83:0/530887049 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f068413ee50 msgr2=0x7f06841da420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:46.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.672+0000 7f068259c640 1 --2- 172.21.15.83:0/530887049 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f068413ee50 0x7f06841da420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:46.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.672+0000 7f068259c640 1 -- 172.21.15.83:0/530887049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0684071480 msgr2=0x7f0684137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:46.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.672+0000 7f068259c640 1 --2- 172.21.15.83:0/530887049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0684071480 0x7f0684137a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:46.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.672+0000 7f068259c640 10 monclient: _finish_hunting 0 2024-03-28T16:14:46.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.672+0000 7f068259c640 1 monclient: found mon.noname-b 2024-03-28T16:14:46.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.672+0000 7f068259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:46.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.672+0000 7f068259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:14:46.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.672+0000 7f068259c640 1 -- 172.21.15.83:0/530887049 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06740094d0 con 0x7f06840b32e0 2024-03-28T16:14:46.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.675+0000 7f068259c640 1 --2- 172.21.15.83:0/530887049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06840b32e0 0x7f0684143c30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f067800c450 tx=0x7f067800c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:46.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.675+0000 7f066b7fe640 10 client.?.objecter ms_handle_connect 0x7f06840b32e0 2024-03-28T16:14:46.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.675+0000 7f066b7fe640 10 client.?.objecter resend_mon_ops 2024-03-28T16:14:46.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f066b7fe640 1 -- 172.21.15.83:0/530887049 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06780052d0 con 0x7f06840b32e0 2024-03-28T16:14:46.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f066b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:46.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f066b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:14:46.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f066b7fe640 10 monclient: dump: 2024-03-28T16:14:46.681 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:46.681 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:46.681 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:46.681 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:46.681 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:46.681 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:46.681 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:46.681 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:46.681 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:46.681 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:46.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f066b7fe640 10 monclient: _finish_auth 0 2024-03-28T16:14:46.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f066b7fe640 10 monclient: _check_auth_tickets 2024-03-28T16:14:46.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f066b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:46.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f066b7fe640 1 -- 172.21.15.83:0/530887049 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0678005430 con 0x7f06840b32e0 2024-03-28T16:14:46.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f068905c640 5 monclient: authenticate success, global_id 4117 2024-03-28T16:14:46.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f068905c640 10 monclient: _renew_subs 2024-03-28T16:14:46.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f068905c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:46.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f068905c640 1 -- 172.21.15.83:0/530887049 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06841445a0 con 0x7f06840b32e0 2024-03-28T16:14:46.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f066b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:46.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f066b7fe640 1 -- 172.21.15.83:0/530887049 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0678012870 con 0x7f06840b32e0 2024-03-28T16:14:46.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f066b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:46.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f066b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:14:46.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f066b7fe640 10 monclient: dump: 2024-03-28T16:14:46.683 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:46.683 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:46.683 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:46.683 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:46.683 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:46.683 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:46.683 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:46.683 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:46.683 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:46.683 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:46.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f068159a640 4 set_mon_vals no callback set 2024-03-28T16:14:46.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f068905c640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:14:46.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f068905c640 10 monclient: _renew_subs 2024-03-28T16:14:46.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f068905c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:46.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f068905c640 1 -- 172.21.15.83:0/530887049 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0684144350 con 0x7f06840b32e0 2024-03-28T16:14:46.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.680+0000 7f068905c640 1 librados: init done 2024-03-28T16:14:46.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.681+0000 7f068905c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:46.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.681+0000 7f068905c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:46.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.681+0000 7f068905c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:46.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.681+0000 7f068905c640 1 -- 172.21.15.83:0/530887049 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06841da100 con 0x7f06840b32e0 2024-03-28T16:14:46.779 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:46.779+0000 7f32d7241040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T16:14:46.816 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:46.815+0000 7f392f430040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T16:14:46.831 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:46.831+0000 7f32d7241040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T16:14:46.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.852+0000 7f066b7fe640 1 -- 172.21.15.83:0/530887049 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0678012cd0 con 0x7f06840b32e0 2024-03-28T16:14:46.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.852+0000 7f066b7fe640 1 -- 172.21.15.83:0/530887049 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0678006cc0 con 0x7f06840b32e0 2024-03-28T16:14:46.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.852+0000 7f066b7fe640 10 client.4117.objecter ms_dispatch 0x7f0684000b90 osd_map(2..2 src has 1..2) v4 2024-03-28T16:14:46.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.852+0000 7f066b7fe640 3 client.4117.objecter handle_osd_map got epochs [2,2] > 0 2024-03-28T16:14:46.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.852+0000 7f066b7fe640 3 client.4117.objecter handle_osd_map decoding full epoch 2 2024-03-28T16:14:46.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.852+0000 7f066b7fe640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-28T16:14:46.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.852+0000 7f066b7fe640 1 -- 172.21.15.83:0/530887049 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f06780035f0 con 0x7f06840b32e0 2024-03-28T16:14:46.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.852+0000 7f066b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:46.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.852+0000 7f066b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:14:46.871 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:46.870+0000 7f392f430040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T16:14:46.877 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:46.877+0000 7f32d7241040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T16:14:46.917 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:46.916+0000 7f392f430040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T16:14:46.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.923+0000 7f068905c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a19c6856-78b9-4919-97ce-6940a5af982c", "id": 0}] 2024-03-28T16:14:46.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.923+0000 7f068905c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a19c6856-78b9-4919-97ce-6940a5af982c", "id": 0}] 2024-03-28T16:14:46.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.923+0000 7f068905c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:46.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:46.923+0000 7f068905c640 1 -- 172.21.15.83:0/530887049 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a19c6856-78b9-4919-97ce-6940a5af982c", "id": 0} v 0) v1 -- 0x7f068413a1c0 con 0x7f06840b32e0 2024-03-28T16:14:46.931 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:46.932+0000 7f32d7241040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T16:14:46.973 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:46.972+0000 7f392f430040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T16:14:46.974 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:46.974+0000 7f32d7241040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T16:14:47.018 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:47.017+0000 7f392f430040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T16:14:47.091 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:47.091+0000 7f32d7241040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T16:14:47.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.097+0000 7f066b7fe640 1 -- 172.21.15.83:0/530887049 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a19c6856-78b9-4919-97ce-6940a5af982c", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f06780070f0 con 0x7f06840b32e0 2024-03-28T16:14:47.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.097+0000 7f066b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a19c6856-78b9-4919-97ce-6940a5af982c", "id": 0}] 2024-03-28T16:14:47.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.097+0000 7f066b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:14:47.097 INFO:teuthology.orchestra.run.smithi083.stdout:0 2024-03-28T16:14:47.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.097+0000 7f068905c640 10 librados: watch_flush enter 2024-03-28T16:14:47.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.097+0000 7f068905c640 10 librados: watch_flush exit 2024-03-28T16:14:47.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.097+0000 7f068905c640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-28T16:14:47.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.097+0000 7f068905c640 10 client.4117.objecter successfully canceled tick 2024-03-28T16:14:47.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.098+0000 7f068905c640 10 monclient: shutdown 2024-03-28T16:14:47.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.098+0000 7f068905c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:47.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.098+0000 7f068905c640 1 -- 172.21.15.83:0/530887049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06840b32e0 msgr2=0x7f0684143c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:47.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.098+0000 7f068905c640 1 --2- 172.21.15.83:0/530887049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06840b32e0 0x7f0684143c30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f067800c450 tx=0x7f067800c910 comp rx=0 tx=0).stop 2024-03-28T16:14:47.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.098+0000 7f068905c640 1 -- 172.21.15.83:0/530887049 shutdown_connections 2024-03-28T16:14:47.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.098+0000 7f068905c640 1 --2- 172.21.15.83:0/530887049 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f068413ee50 0x7f06841da420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.098+0000 7f068905c640 1 --2- 172.21.15.83:0/530887049 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06840b32e0 0x7f0684143c30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.098+0000 7f068905c640 1 --2- 172.21.15.83:0/530887049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0684071480 0x7f0684137a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.098+0000 7f068905c640 1 -- 172.21.15.83:0/530887049 >> 172.21.15.83:0/530887049 conn(0x7f06840b70e0 msgr2=0x7f0684138fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:47.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.098+0000 7f068905c640 1 -- 172.21.15.83:0/530887049 shutdown_connections 2024-03-28T16:14:47.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.098+0000 7f068905c640 1 -- 172.21.15.83:0/530887049 wait complete. 2024-03-28T16:14:47.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.098+0000 7f068905c640 1 librados: shutdown 2024-03-28T16:14:47.108 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd new 2afee73d-3369-4fd2-bfc5-36dee610daf2 1 2024-03-28T16:14:47.159 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:47.158+0000 7f392f430040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T16:14:47.166 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:47.167+0000 7f32d7241040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T16:14:47.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.207+0000 7f8ee037e640 10 monclient: get_monmap_and_config 2024-03-28T16:14:47.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.207+0000 7f8ee037e640 10 monclient: build_initial_monmap 2024-03-28T16:14:47.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.207+0000 7f8ee037e640 10 monclient: monmap: 2024-03-28T16:14:47.207 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-28T16:14:47.207 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:47.207 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:14:47.207862+0000 2024-03-28T16:14:47.207 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:14:47.207862+0000 2024-03-28T16:14:47.207 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:47.207 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:47.207 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:14:47.208 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:14:47.208 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:14:47.208 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:14:47.208 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:14:47.208 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:47.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.207+0000 7f8ee037e640 1 Processor -- start 2024-03-28T16:14:47.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.208+0000 7f8ee037e640 1 -- start start 2024-03-28T16:14:47.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.208+0000 7f8ee037e640 10 monclient: init 2024-03-28T16:14:47.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.208+0000 7f8ee037e640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:47.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.208+0000 7f8ee037e640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-03-28T16:14:47.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.208+0000 7f8ee037e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8ed8071920 0x7f8ed813e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:47.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.208+0000 7f8ee037e640 10 monclient(hunting): picked mon.noname-b con 0x7f8ed8071920 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:47.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.208+0000 7f8ee037e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8ed80a89a0 addr v1:172.21.15.83:6789/0 2024-03-28T16:14:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.208+0000 7f8ee037e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8ed813ed70 0x7f8ed80b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.208+0000 7f8ee037e640 10 monclient(hunting): picked mon.noname-a con 0x7f8ed813ed70 addr v2:172.21.15.7:3300/0 2024-03-28T16:14:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.208+0000 7f8ee037e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed80a8ec0 con 0x7f8ed80a89a0 2024-03-28T16:14:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.208+0000 7f8ee037e640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.208+0000 7f8ee037e640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ed80aa480 con 0x7f8ed813ed70 2024-03-28T16:14:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.208+0000 7f8ee037e640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.208+0000 7f8ee037e640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed813f200 con 0x7f8ed8071920 2024-03-28T16:14:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.208+0000 7f8ee037e640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.208+0000 7f8ee037e640 10 monclient(hunting): authenticate will time out at 1354.377726s 2024-03-28T16:14:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.209+0000 7f8ede91b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8ed813ed70 0x7f8ed80b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.209+0000 7f8edd919640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8ed80a89a0 0x7f8ed80b1890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60882/0 (socket says 172.21.15.83:60882) 2024-03-28T16:14:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.209+0000 7f8edd919640 1 -- 172.21.15.83:0/2231530469 learned_addr learned my addr 172.21.15.83:0/2231530469 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-28T16:14:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.209+0000 7f8ede91b640 10 monclient(hunting): get_auth_request con 0x7f8ed813ed70 auth_method 0 2024-03-28T16:14:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.209+0000 7f8ede91b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.209+0000 7f8ede91b640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:47.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.209+0000 7f8ede91b640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:47.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.210+0000 7f8ede91b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:47.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.210+0000 7f8ede91b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:47.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.210+0000 7f8ede91b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:47.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.210+0000 7f8ede11a640 1 --2- 172.21.15.83:0/2231530469 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8ed8071920 0x7f8ed813e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:47.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.210+0000 7f8ede11a640 10 monclient(hunting): get_auth_request con 0x7f8ed8071920 auth_method 0 2024-03-28T16:14:47.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.210+0000 7f8ede11a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:47.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.210+0000 7f8ede11a640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:47.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.210+0000 7f8ede11a640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:47.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.211+0000 7f8ede11a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:47.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.211+0000 7f8ede11a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:47.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.211+0000 7f8ede11a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:47.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.212+0000 7f8edd118640 1 -- 172.21.15.83:0/2231530469 <== mon.1 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111499230 0 0) 0x7f8ed80a8ec0 con 0x7f8ed80a89a0 2024-03-28T16:14:47.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.212+0000 7f8edd118640 10 monclient(hunting): _negotiate 2024-03-28T16:14:47.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.212+0000 7f8edd118640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:47.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.212+0000 7f8edd118640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:47.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.212+0000 7f8edd118640 10 monclient(hunting): my global_id is 4120 2024-03-28T16:14:47.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.212+0000 7f8edd118640 1 -- 172.21.15.83:0/2231530469 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ec4003520 con 0x7f8ed80a89a0 2024-03-28T16:14:47.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.212+0000 7f8ede91b640 10 monclient(hunting): handle_auth_done global_id 4119 payload 274 2024-03-28T16:14:47.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.212+0000 7f8ede91b640 1 -- 172.21.15.83:0/2231530469 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8ed8071920 msgr2=0x7f8ed813e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:14:47.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.212+0000 7f8ede91b640 1 --2- 172.21.15.83:0/2231530469 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8ed8071920 0x7f8ed813e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.212+0000 7f8ede91b640 1 -- 172.21.15.83:0/2231530469 >> v1:172.21.15.83:6789/0 conn(0x7f8ed80a89a0 legacy=0x7f8ed80b1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:47.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.212+0000 7f8ede91b640 10 monclient: _finish_hunting 0 2024-03-28T16:14:47.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.212+0000 7f8ede91b640 1 monclient: found mon.noname-a 2024-03-28T16:14:47.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.212+0000 7f8ede91b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:47.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.212+0000 7f8ede91b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:14:47.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.212+0000 7f8ede91b640 1 -- 172.21.15.83:0/2231530469 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed80aa660 con 0x7f8ed813ed70 2024-03-28T16:14:47.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.215+0000 7f8ede91b640 1 --2- 172.21.15.83:0/2231530469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed813ed70 0x7f8ed80b1fa0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0009d90 tx=0x7f8ed002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a51c222d1fa19e8f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:47.215 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:47.215+0000 7f32d7241040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T16:14:47.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.218+0000 7f8edd118640 1 -- 172.21.15.83:0/2231530469 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed002cbc0 con 0x7f8ed813ed70 2024-03-28T16:14:47.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.218+0000 7f8edd118640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:47.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8edd118640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:14:47.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8edd118640 10 monclient: dump: 2024-03-28T16:14:47.219 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:47.219 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:47.219 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.219 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.219 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:47.219 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:47.219 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:47.219 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:47.219 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:47.220 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:47.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8edd118640 10 monclient: _finish_auth 0 2024-03-28T16:14:47.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8edd118640 10 monclient: _check_auth_tickets 2024-03-28T16:14:47.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8edd118640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:47.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8edd118640 1 -- 172.21.15.83:0/2231530469 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed002cd20 con 0x7f8ed813ed70 2024-03-28T16:14:47.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8ee037e640 5 monclient: authenticate success, global_id 4119 2024-03-28T16:14:47.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8ee037e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:14:47.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8edd118640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:47.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8ee037e640 10 monclient: get_monmap_and_config success 2024-03-28T16:14:47.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8ee037e640 4 set_mon_vals no callback set 2024-03-28T16:14:47.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8edd118640 1 -- 172.21.15.83:0/2231530469 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed0034660 con 0x7f8ed813ed70 2024-03-28T16:14:47.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8edd118640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:47.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8edd118640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:14:47.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8edd118640 10 monclient: dump: 2024-03-28T16:14:47.221 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:47.221 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:47.221 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.221 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.221 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:47.221 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:47.221 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:47.221 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:47.221 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:47.221 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:47.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8ee037e640 10 monclient: shutdown 2024-03-28T16:14:47.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8ee037e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:47.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8ee037e640 1 -- 172.21.15.83:0/2231530469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed813ed70 msgr2=0x7f8ed80b1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:47.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8ee037e640 1 --2- 172.21.15.83:0/2231530469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed813ed70 0x7f8ed80b1fa0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0009d90 tx=0x7f8ed002be10 comp rx=0 tx=0).stop 2024-03-28T16:14:47.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8ee037e640 1 -- 172.21.15.83:0/2231530469 shutdown_connections 2024-03-28T16:14:47.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8ee037e640 1 --2- 172.21.15.83:0/2231530469 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed813ed70 0x7f8ed80b1fa0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8ee037e640 1 --2- 172.21.15.83:0/2231530469 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8ed8071920 0x7f8ed813e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8ee037e640 1 -- 172.21.15.83:0/2231530469 >> 172.21.15.83:0/2231530469 conn(0x7f8ed80b3db0 msgr2=0x7f8ed80b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:47.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8ee037e640 1 -- 172.21.15.83:0/2231530469 shutdown_connections 2024-03-28T16:14:47.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.219+0000 7f8ee037e640 1 -- 172.21.15.83:0/2231530469 wait complete. 2024-03-28T16:14:47.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.220+0000 7f8ee037e640 10 monclient: build_initial_monmap 2024-03-28T16:14:47.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.220+0000 7f8ee037e640 10 monclient: monmap: 2024-03-28T16:14:47.225 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-28T16:14:47.225 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:14:47.225 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-28T16:14:47.225 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-28T16:14:47.225 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:47.226 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:47.226 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:14:47.226 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:14:47.226 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:14:47.226 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:47.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.220+0000 7f8ee037e640 1 librados: starting msgr at 2024-03-28T16:14:47.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.220+0000 7f8ee037e640 1 librados: starting objecter 2024-03-28T16:14:47.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.220+0000 7f8ee037e640 1 Processor -- start 2024-03-28T16:14:47.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.220+0000 7f8ee037e640 1 -- start start 2024-03-28T16:14:47.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.220+0000 7f8ee037e640 1 librados: setting wanted keys 2024-03-28T16:14:47.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.220+0000 7f8ee037e640 1 librados: calling monclient init 2024-03-28T16:14:47.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.220+0000 7f8ee037e640 10 monclient: init 2024-03-28T16:14:47.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.221+0000 7f8ee037e640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:47.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.221+0000 7f8ee037e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-28T16:14:47.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.221+0000 7f8ee037e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed8071920 0x7f8ed81375c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:47.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.221+0000 7f8ee037e640 10 monclient(hunting): picked mon.noname-a con 0x7f8ed8071920 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:14:47.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.221+0000 7f8ee037e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8ed813ed70 0x7f8ed8131980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:47.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.221+0000 7f8ee037e640 10 monclient(hunting): picked mon.noname-c con 0x7f8ed813ed70 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:47.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.221+0000 7f8ee037e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ed81390b0 0x7f8ed8131ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:47.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.221+0000 7f8ee037e640 10 monclient(hunting): picked mon.noname-b con 0x7f8ed81390b0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:14:47.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.221+0000 7f8ee037e640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:47.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.221+0000 7f8ee037e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed8145920 con 0x7f8ed8071920 2024-03-28T16:14:47.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.221+0000 7f8ee037e640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:47.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.221+0000 7f8ee037e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed81389b0 con 0x7f8ed81390b0 2024-03-28T16:14:47.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.221+0000 7f8ee037e640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:47.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.221+0000 7f8ee037e640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed81440c0 con 0x7f8ed813ed70 2024-03-28T16:14:47.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.221+0000 7f8ee037e640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:47.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.221+0000 7f8ee037e640 10 monclient(hunting): authenticate will time out at 1354.390362s 2024-03-28T16:14:47.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.221+0000 7f8ede91b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ed81390b0 0x7f8ed8131ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:47.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.222+0000 7f8edd919640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8ed813ed70 0x7f8ed8131980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:47.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.222+0000 7f8ede11a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed8071920 0x7f8ed81375c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:47.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.222+0000 7f8ede91b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ed81390b0 0x7f8ed8131ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35970/0 (socket says 172.21.15.83:35970) 2024-03-28T16:14:47.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.222+0000 7f8ede91b640 1 -- 172.21.15.83:0/1085698946 learned_addr learned my addr 172.21.15.83:0/1085698946 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-28T16:14:47.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.222+0000 7f8ede91b640 10 monclient(hunting): get_auth_request con 0x7f8ed81390b0 auth_method 0 2024-03-28T16:14:47.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.222+0000 7f8ede91b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:47.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.222+0000 7f8ede91b640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:47.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.222+0000 7f8ede91b640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:47.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.222+0000 7f8ede11a640 10 monclient(hunting): get_auth_request con 0x7f8ed8071920 auth_method 0 2024-03-28T16:14:47.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.222+0000 7f8ede11a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:47.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.222+0000 7f8ede11a640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:47.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.222+0000 7f8ede11a640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:47.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.222+0000 7f8edd919640 10 monclient(hunting): get_auth_request con 0x7f8ed813ed70 auth_method 0 2024-03-28T16:14:47.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.222+0000 7f8edd919640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:47.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.222+0000 7f8edd919640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:47.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.222+0000 7f8edd919640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:47.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.222+0000 7f8ede91b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:47.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.222+0000 7f8ede91b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:47.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.222+0000 7f8ede91b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:47.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.223+0000 7f8edd919640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:47.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.223+0000 7f8edd919640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:47.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.223+0000 7f8edd919640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:47.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.223+0000 7f8ede11a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:47.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.223+0000 7f8ede11a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:47.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.223+0000 7f8ede11a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:47.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.225+0000 7f8ede91b640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2024-03-28T16:14:47.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.225+0000 7f8ede91b640 1 -- 172.21.15.83:0/1085698946 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8ed813ed70 msgr2=0x7f8ed8131980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:47.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.225+0000 7f8ede91b640 1 --2- 172.21.15.83:0/1085698946 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8ed813ed70 0x7f8ed8131980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.225+0000 7f8ede91b640 1 -- 172.21.15.83:0/1085698946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed8071920 msgr2=0x7f8ed81375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:47.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.225+0000 7f8ede91b640 1 --2- 172.21.15.83:0/1085698946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed8071920 0x7f8ed81375c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.225+0000 7f8ede91b640 10 monclient: _finish_hunting 0 2024-03-28T16:14:47.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.225+0000 7f8ede91b640 1 monclient: found mon.noname-b 2024-03-28T16:14:47.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.225+0000 7f8ede91b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:47.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.225+0000 7f8ede91b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:14:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.225+0000 7f8ede91b640 1 -- 172.21.15.83:0/1085698946 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed0009a70 con 0x7f8ed81390b0 2024-03-28T16:14:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.227+0000 7f8ede91b640 1 --2- 172.21.15.83:0/1085698946 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ed81390b0 0x7f8ed8131ec0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0009eb0 tx=0x7f8ed0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.227+0000 7f8eceffd640 10 client.?.objecter ms_handle_connect 0x7f8ed81390b0 2024-03-28T16:14:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.227+0000 7f8eceffd640 10 client.?.objecter resend_mon_ops 2024-03-28T16:14:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.229+0000 7f8eceffd640 1 -- 172.21.15.83:0/1085698946 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed003e610 con 0x7f8ed81390b0 2024-03-28T16:14:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.229+0000 7f8eceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.229+0000 7f8eceffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:14:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.229+0000 7f8eceffd640 10 monclient: dump: 2024-03-28T16:14:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:47.232 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:47.233 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:47.233 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:47.233 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:47.233 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:47.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.229+0000 7f8eceffd640 10 monclient: _finish_auth 0 2024-03-28T16:14:47.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.229+0000 7f8eceffd640 10 monclient: _check_auth_tickets 2024-03-28T16:14:47.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.229+0000 7f8eceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:47.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.229+0000 7f8ee037e640 5 monclient: authenticate success, global_id 4123 2024-03-28T16:14:47.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.229+0000 7f8ee037e640 10 monclient: _renew_subs 2024-03-28T16:14:47.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.229+0000 7f8ee037e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:47.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.229+0000 7f8ee037e640 1 -- 172.21.15.83:0/1085698946 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed8137b00 con 0x7f8ed81390b0 2024-03-28T16:14:47.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.229+0000 7f8ee037e640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:14:47.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.229+0000 7f8ee037e640 10 monclient: _renew_subs 2024-03-28T16:14:47.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.229+0000 7f8ee037e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:47.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.229+0000 7f8ee037e640 1 -- 172.21.15.83:0/1085698946 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed8145210 con 0x7f8ed81390b0 2024-03-28T16:14:47.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.229+0000 7f8ee037e640 1 librados: init done 2024-03-28T16:14:47.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.230+0000 7f8ee037e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:47.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.230+0000 7f8ee037e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:47.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.230+0000 7f8ee037e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:47.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.230+0000 7f8ee037e640 1 -- 172.21.15.83:0/1085698946 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed8069430 con 0x7f8ed81390b0 2024-03-28T16:14:47.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.231+0000 7f8eceffd640 1 -- 172.21.15.83:0/1085698946 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed002ce90 con 0x7f8ed81390b0 2024-03-28T16:14:47.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.231+0000 7f8eceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:47.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.231+0000 7f8eceffd640 1 -- 172.21.15.83:0/1085698946 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed003e970 con 0x7f8ed81390b0 2024-03-28T16:14:47.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.231+0000 7f8eceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:47.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.231+0000 7f8eceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:14:47.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.231+0000 7f8eceffd640 10 monclient: dump: 2024-03-28T16:14:47.234 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:47.234 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:47.234 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.235 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.235 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:47.235 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:47.235 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:47.235 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:47.235 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:47.235 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:47.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.231+0000 7f8ecffff640 4 set_mon_vals no callback set 2024-03-28T16:14:47.235 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:47.232+0000 7f392f430040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T16:14:47.270 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:47.271+0000 7f32d7241040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T16:14:47.282 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:47.281+0000 7f392f430040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T16:14:47.318 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:47.318+0000 7f32d7241040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T16:14:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.333+0000 7f8eceffd640 1 -- 172.21.15.83:0/1085698946 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8ed003b050 con 0x7f8ed81390b0 2024-03-28T16:14:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.333+0000 7f8eceffd640 1 -- 172.21.15.83:0/1085698946 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f8ed00499c0 con 0x7f8ed81390b0 2024-03-28T16:14:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.333+0000 7f8eceffd640 10 client.4123.objecter ms_dispatch 0x7f8ed8000b90 osd_map(3..3 src has 1..3) v4 2024-03-28T16:14:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.333+0000 7f8eceffd640 3 client.4123.objecter handle_osd_map got epochs [3,3] > 0 2024-03-28T16:14:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.333+0000 7f8eceffd640 3 client.4123.objecter handle_osd_map decoding full epoch 3 2024-03-28T16:14:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.334+0000 7f8eceffd640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-28T16:14:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.334+0000 7f8eceffd640 1 -- 172.21.15.83:0/1085698946 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f8ed004e060 con 0x7f8ed81390b0 2024-03-28T16:14:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.334+0000 7f8eceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:47.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.334+0000 7f8eceffd640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:14:47.338 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:47.337+0000 7f392f430040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T16:14:47.390 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:47.389+0000 7f392f430040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T16:14:47.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.409+0000 7f8ee037e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2afee73d-3369-4fd2-bfc5-36dee610daf2", "id": 1}] 2024-03-28T16:14:47.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.409+0000 7f8ee037e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2afee73d-3369-4fd2-bfc5-36dee610daf2", "id": 1}] 2024-03-28T16:14:47.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.409+0000 7f8ee037e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:47.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.409+0000 7f8ee037e640 1 -- 172.21.15.83:0/1085698946 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2afee73d-3369-4fd2-bfc5-36dee610daf2", "id": 1} v 0) v1 -- 0x7f8ed813d840 con 0x7f8ed81390b0 2024-03-28T16:14:47.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.486+0000 7f8eceffd640 1 -- 172.21.15.83:0/1085698946 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2afee73d-3369-4fd2-bfc5-36dee610daf2", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8ed00560b0 con 0x7f8ed81390b0 2024-03-28T16:14:47.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.486+0000 7f8eceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2afee73d-3369-4fd2-bfc5-36dee610daf2", "id": 1}] 2024-03-28T16:14:47.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.486+0000 7f8eceffd640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:14:47.487 INFO:teuthology.orchestra.run.smithi083.stdout:1 2024-03-28T16:14:47.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.487+0000 7f8ee037e640 10 librados: watch_flush enter 2024-03-28T16:14:47.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.487+0000 7f8ee037e640 10 librados: watch_flush exit 2024-03-28T16:14:47.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.487+0000 7f8ee037e640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-28T16:14:47.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.487+0000 7f8ee037e640 10 client.4123.objecter successfully canceled tick 2024-03-28T16:14:47.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.487+0000 7f8ee037e640 10 monclient: shutdown 2024-03-28T16:14:47.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.487+0000 7f8ee037e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:47.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.487+0000 7f8ee037e640 1 -- 172.21.15.83:0/1085698946 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ed81390b0 msgr2=0x7f8ed8131ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:47.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.487+0000 7f8ee037e640 1 --2- 172.21.15.83:0/1085698946 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ed81390b0 0x7f8ed8131ec0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0009eb0 tx=0x7f8ed0004a40 comp rx=0 tx=0).stop 2024-03-28T16:14:47.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.487+0000 7f8ee037e640 1 -- 172.21.15.83:0/1085698946 shutdown_connections 2024-03-28T16:14:47.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.487+0000 7f8ee037e640 1 --2- 172.21.15.83:0/1085698946 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ed81390b0 0x7f8ed8131ec0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.487+0000 7f8ee037e640 1 --2- 172.21.15.83:0/1085698946 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8ed813ed70 0x7f8ed8131980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.487+0000 7f8ee037e640 1 --2- 172.21.15.83:0/1085698946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed8071920 0x7f8ed81375c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.487+0000 7f8ee037e640 1 -- 172.21.15.83:0/1085698946 >> 172.21.15.83:0/1085698946 conn(0x7f8ed80b3db0 msgr2=0x7f8ed80b1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:47.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.487+0000 7f8ee037e640 1 -- 172.21.15.83:0/1085698946 shutdown_connections 2024-03-28T16:14:47.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.487+0000 7f8ee037e640 1 -- 172.21.15.83:0/1085698946 wait complete. 2024-03-28T16:14:47.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.487+0000 7f8ee037e640 1 librados: shutdown 2024-03-28T16:14:47.497 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd new 0473d84c-e864-4373-b9d1-283130278363 2 2024-03-28T16:14:47.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.597+0000 7f8f8c7ce640 10 monclient: get_monmap_and_config 2024-03-28T16:14:47.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.597+0000 7f8f8c7ce640 10 monclient: build_initial_monmap 2024-03-28T16:14:47.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.597+0000 7f8f8c7ce640 10 monclient: monmap: 2024-03-28T16:14:47.598 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-28T16:14:47.598 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:47.598 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:14:47.598188+0000 2024-03-28T16:14:47.598 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:14:47.598188+0000 2024-03-28T16:14:47.598 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:47.598 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:47.598 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:14:47.598 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:14:47.598 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:14:47.599 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:14:47.599 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:14:47.599 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:47.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.598+0000 7f8f8c7ce640 1 Processor -- start 2024-03-28T16:14:47.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.598+0000 7f8f8c7ce640 1 -- start start 2024-03-28T16:14:47.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.598+0000 7f8f8c7ce640 10 monclient: init 2024-03-28T16:14:47.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.598+0000 7f8f8c7ce640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:47.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.598+0000 7f8f8c7ce640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-03-28T16:14:47.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.598+0000 7f8f8c7ce640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8f84071480 0x7f8f8413e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:47.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f8c7ce640 10 monclient(hunting): picked mon.noname-a con 0x7f8f84071480 addr v2:172.21.15.7:3300/0 2024-03-28T16:14:47.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f8c7ce640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8f840b32e0 0x7f8f84144a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:47.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f8c7ce640 10 monclient(hunting): picked mon.noname-c con 0x7f8f840b32e0 addr v2:172.21.15.83:3300/0 2024-03-28T16:14:47.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f8c7ce640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f8413ee50 0x7f8f84144f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:47.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f8c7ce640 10 monclient(hunting): picked mon.noname-b con 0x7f8f8413ee50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:47.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f8c7ce640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:47.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f8c7ce640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f84071860 con 0x7f8f84071480 2024-03-28T16:14:47.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f8c7ce640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:47.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f8c7ce640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f840a8c90 con 0x7f8f840b32e0 2024-03-28T16:14:47.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f8c7ce640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:47.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f8c7ce640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f84072140 con 0x7f8f8413ee50 2024-03-28T16:14:47.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f8c7ce640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:47.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f8c7ce640 10 monclient(hunting): authenticate will time out at 1354.767920s 2024-03-28T16:14:47.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f89d69640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8f840b32e0 0x7f8f84144a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:47.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f8ad6b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f8413ee50 0x7f8f84144f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:47.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f8a56a640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8f84071480 0x7f8f8413e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:47.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f89d69640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f8f840b32e0 0x7f8f84144a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35986/0 (socket says 172.21.15.83:35986) 2024-03-28T16:14:47.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f89d69640 1 -- 172.21.15.83:0/2757693323 learned_addr learned my addr 172.21.15.83:0/2757693323 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-28T16:14:47.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f89d69640 10 monclient(hunting): get_auth_request con 0x7f8f840b32e0 auth_method 0 2024-03-28T16:14:47.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f89d69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:47.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f89d69640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:47.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.599+0000 7f8f89d69640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:47.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.600+0000 7f8f8a56a640 10 monclient(hunting): get_auth_request con 0x7f8f84071480 auth_method 0 2024-03-28T16:14:47.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.600+0000 7f8f8a56a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:47.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.600+0000 7f8f8a56a640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:47.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.600+0000 7f8f8a56a640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:47.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.600+0000 7f8f8ad6b640 10 monclient(hunting): get_auth_request con 0x7f8f8413ee50 auth_method 0 2024-03-28T16:14:47.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.600+0000 7f8f8ad6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:47.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.600+0000 7f8f8ad6b640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:47.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.600+0000 7f8f8ad6b640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:47.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.600+0000 7f8f89d69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:47.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.600+0000 7f8f89d69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:47.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.600+0000 7f8f89d69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:47.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.600+0000 7f8f8a56a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:47.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.600+0000 7f8f8a56a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:47.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.600+0000 7f8f8a56a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:47.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.600+0000 7f8f8ad6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:47.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.600+0000 7f8f8ad6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:47.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.600+0000 7f8f8ad6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:47.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.602+0000 7f8f89d69640 10 monclient(hunting): handle_auth_done global_id 4126 payload 274 2024-03-28T16:14:47.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.602+0000 7f8f89d69640 1 -- 172.21.15.83:0/2757693323 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f8413ee50 msgr2=0x7f8f84144f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:14:47.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.602+0000 7f8f89d69640 1 --2- 172.21.15.83:0/2757693323 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f8413ee50 0x7f8f84144f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.602+0000 7f8f89d69640 1 -- 172.21.15.83:0/2757693323 >> v2:172.21.15.7:3300/0 conn(0x7f8f84071480 msgr2=0x7f8f8413e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:14:47.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.602+0000 7f8f89d69640 1 --2- 172.21.15.83:0/2757693323 >> v2:172.21.15.7:3300/0 conn(0x7f8f84071480 0x7f8f8413e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.602+0000 7f8f89d69640 10 monclient: _finish_hunting 0 2024-03-28T16:14:47.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.602+0000 7f8f89d69640 1 monclient: found mon.noname-c 2024-03-28T16:14:47.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.602+0000 7f8f89d69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:47.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.602+0000 7f8f89d69640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3300/0 2024-03-28T16:14:47.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.602+0000 7f8f89d69640 1 -- 172.21.15.83:0/2757693323 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f840722b0 con 0x7f8f840b32e0 2024-03-28T16:14:47.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.603+0000 7f8f89d69640 1 --2- 172.21.15.83:0/2757693323 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f840b32e0 0x7f8f84144a00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8f84071ac0 tx=0x7f8f7402b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e2d3a50bdec6a26 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:47.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.607+0000 7f8f89568640 1 -- 172.21.15.83:0/2757693323 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f74031750 con 0x7f8f840b32e0 2024-03-28T16:14:47.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.607+0000 7f8f89568640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:47.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.607+0000 7f8f89568640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:14:47.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.607+0000 7f8f89568640 10 monclient: dump: 2024-03-28T16:14:47.607 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:47.607 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:47.608 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.608 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.608 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:47.608 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:47.608 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:47.608 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:47.608 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:47.608 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:47.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.607+0000 7f8f89568640 10 monclient: _finish_auth 0 2024-03-28T16:14:47.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.607+0000 7f8f89568640 10 monclient: _check_auth_tickets 2024-03-28T16:14:47.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.607+0000 7f8f89568640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:47.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.607+0000 7f8f89568640 1 -- 172.21.15.83:0/2757693323 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f740318b0 con 0x7f8f840b32e0 2024-03-28T16:14:47.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.607+0000 7f8f89568640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:47.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.607+0000 7f8f89568640 1 -- 172.21.15.83:0/2757693323 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f74031c40 con 0x7f8f840b32e0 2024-03-28T16:14:47.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.607+0000 7f8f8c7ce640 5 monclient: authenticate success, global_id 4126 2024-03-28T16:14:47.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.607+0000 7f8f8c7ce640 10 monclient: get_monmap_and_config success 2024-03-28T16:14:47.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.607+0000 7f8f8c7ce640 4 set_mon_vals no callback set 2024-03-28T16:14:47.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.607+0000 7f8f89568640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:47.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.607+0000 7f8f89568640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:14:47.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.607+0000 7f8f89568640 10 monclient: dump: 2024-03-28T16:14:47.609 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:47.609 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:47.609 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.609 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.609 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:47.609 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:47.609 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:47.609 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:47.609 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:47.609 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:47.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.608+0000 7f8f8c7ce640 10 monclient: shutdown 2024-03-28T16:14:47.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.608+0000 7f8f8c7ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:47.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.608+0000 7f8f8c7ce640 1 -- 172.21.15.83:0/2757693323 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f840b32e0 msgr2=0x7f8f84144a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:47.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.608+0000 7f8f8c7ce640 1 --2- 172.21.15.83:0/2757693323 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f840b32e0 0x7f8f84144a00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8f84071ac0 tx=0x7f8f7402b7f0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.608+0000 7f8f8c7ce640 1 -- 172.21.15.83:0/2757693323 shutdown_connections 2024-03-28T16:14:47.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.608+0000 7f8f8c7ce640 1 --2- 172.21.15.83:0/2757693323 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f8413ee50 0x7f8f84144f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.608+0000 7f8f8c7ce640 1 --2- 172.21.15.83:0/2757693323 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f840b32e0 0x7f8f84144a00 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.608+0000 7f8f8c7ce640 1 --2- 172.21.15.83:0/2757693323 >> v2:172.21.15.7:3300/0 conn(0x7f8f84071480 0x7f8f8413e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.608+0000 7f8f8c7ce640 1 -- 172.21.15.83:0/2757693323 >> 172.21.15.83:0/2757693323 conn(0x7f8f840b70e0 msgr2=0x7f8f840b74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:47.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.608+0000 7f8f8c7ce640 1 -- 172.21.15.83:0/2757693323 shutdown_connections 2024-03-28T16:14:47.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.608+0000 7f8f8c7ce640 1 -- 172.21.15.83:0/2757693323 wait complete. 2024-03-28T16:14:47.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.609+0000 7f8f8c7ce640 10 monclient: build_initial_monmap 2024-03-28T16:14:47.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.609+0000 7f8f8c7ce640 10 monclient: monmap: 2024-03-28T16:14:47.613 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-28T16:14:47.613 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:14:47.614 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-28T16:14:47.614 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-28T16:14:47.614 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:47.614 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:47.614 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:14:47.614 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:14:47.614 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:14:47.614 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:47.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.609+0000 7f8f8c7ce640 1 librados: starting msgr at 2024-03-28T16:14:47.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.609+0000 7f8f8c7ce640 1 librados: starting objecter 2024-03-28T16:14:47.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.609+0000 7f8f8c7ce640 1 Processor -- start 2024-03-28T16:14:47.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.609+0000 7f8f8c7ce640 1 -- start start 2024-03-28T16:14:47.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.609+0000 7f8f8c7ce640 1 librados: setting wanted keys 2024-03-28T16:14:47.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.609+0000 7f8f8c7ce640 1 librados: calling monclient init 2024-03-28T16:14:47.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.609+0000 7f8f8c7ce640 10 monclient: init 2024-03-28T16:14:47.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f8c7ce640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:47.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f8c7ce640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-28T16:14:47.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f8c7ce640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f84071480 0x7f8f8413fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:47.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f8c7ce640 10 monclient(hunting): picked mon.noname-a con 0x7f8f84071480 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:14:47.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f8c7ce640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f840b32e0 0x7f8f84140aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:47.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f8c7ce640 10 monclient(hunting): picked mon.noname-b con 0x7f8f840b32e0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:14:47.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f8c7ce640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f8413ee50 0x7f8f841da030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:47.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f8c7ce640 10 monclient(hunting): picked mon.noname-c con 0x7f8f8413ee50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:47.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f8c7ce640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:47.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f8c7ce640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f84140fe0 con 0x7f8f84071480 2024-03-28T16:14:47.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f8c7ce640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:47.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f8c7ce640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8413f330 con 0x7f8f840b32e0 2024-03-28T16:14:47.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f8c7ce640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:47.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f8c7ce640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8413e050 con 0x7f8f8413ee50 2024-03-28T16:14:47.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f8c7ce640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:47.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f8c7ce640 10 monclient(hunting): authenticate will time out at 1354.779034s 2024-03-28T16:14:47.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f89d69640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f840b32e0 0x7f8f84140aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:47.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f8ad6b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f8413ee50 0x7f8f841da030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:47.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f8a56a640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f84071480 0x7f8f8413fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:47.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f89d69640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f840b32e0 0x7f8f84140aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35988/0 (socket says 172.21.15.83:35988) 2024-03-28T16:14:47.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f89d69640 1 -- 172.21.15.83:0/3759860932 learned_addr learned my addr 172.21.15.83:0/3759860932 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-28T16:14:47.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f89d69640 10 monclient(hunting): get_auth_request con 0x7f8f840b32e0 auth_method 0 2024-03-28T16:14:47.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f89d69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:47.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f89d69640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:47.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.610+0000 7f8f89d69640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:47.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.611+0000 7f8f8ad6b640 10 monclient(hunting): get_auth_request con 0x7f8f8413ee50 auth_method 0 2024-03-28T16:14:47.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.611+0000 7f8f8ad6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:47.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.611+0000 7f8f8ad6b640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:47.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.611+0000 7f8f8ad6b640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:47.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.611+0000 7f8f8a56a640 10 monclient(hunting): get_auth_request con 0x7f8f84071480 auth_method 0 2024-03-28T16:14:47.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.611+0000 7f8f8a56a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:47.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.611+0000 7f8f8a56a640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:47.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.611+0000 7f8f8a56a640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:47.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.611+0000 7f8f89d69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:47.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.611+0000 7f8f89d69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:47.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.611+0000 7f8f89d69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:47.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.611+0000 7f8f8ad6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:47.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.611+0000 7f8f8ad6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:47.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.611+0000 7f8f8ad6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:47.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.611+0000 7f8f8a56a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:47.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.611+0000 7f8f8a56a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:47.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.611+0000 7f8f8a56a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:47.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.614+0000 7f8f89d69640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2024-03-28T16:14:47.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.614+0000 7f8f89d69640 1 -- 172.21.15.83:0/3759860932 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f8413ee50 msgr2=0x7f8f841da030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:47.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.614+0000 7f8f89d69640 1 --2- 172.21.15.83:0/3759860932 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f8413ee50 0x7f8f841da030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.614+0000 7f8f89d69640 1 -- 172.21.15.83:0/3759860932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f84071480 msgr2=0x7f8f8413fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:47.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.614+0000 7f8f89d69640 1 --2- 172.21.15.83:0/3759860932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f84071480 0x7f8f8413fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.614+0000 7f8f89d69640 10 monclient: _finish_hunting 0 2024-03-28T16:14:47.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.614+0000 7f8f89d69640 1 monclient: found mon.noname-b 2024-03-28T16:14:47.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.614+0000 7f8f89d69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:47.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.614+0000 7f8f89d69640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:14:47.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.614+0000 7f8f89d69640 1 -- 172.21.15.83:0/3759860932 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f740094d0 con 0x7f8f840b32e0 2024-03-28T16:14:47.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.616+0000 7f8f89d69640 1 --2- 172.21.15.83:0/3759860932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f840b32e0 0x7f8f84140aa0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8f7402b830 tx=0x7f8f740046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:47.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.616+0000 7f8f7affd640 10 client.?.objecter ms_handle_connect 0x7f8f840b32e0 2024-03-28T16:14:47.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.616+0000 7f8f7affd640 10 client.?.objecter resend_mon_ops 2024-03-28T16:14:47.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f7affd640 1 -- 172.21.15.83:0/3759860932 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f740338d0 con 0x7f8f840b32e0 2024-03-28T16:14:47.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:47.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f7affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:14:47.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f7affd640 10 monclient: dump: 2024-03-28T16:14:47.621 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:47.621 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:47.621 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.621 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.621 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:47.621 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:47.621 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:47.622 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:47.622 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:47.622 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:47.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f7affd640 10 monclient: _finish_auth 0 2024-03-28T16:14:47.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f7affd640 10 monclient: _check_auth_tickets 2024-03-28T16:14:47.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:47.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f7affd640 1 -- 172.21.15.83:0/3759860932 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f74033a30 con 0x7f8f840b32e0 2024-03-28T16:14:47.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f8c7ce640 5 monclient: authenticate success, global_id 4129 2024-03-28T16:14:47.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f8c7ce640 10 monclient: _renew_subs 2024-03-28T16:14:47.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f8c7ce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:47.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f8c7ce640 1 -- 172.21.15.83:0/3759860932 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f841d9b10 con 0x7f8f840b32e0 2024-03-28T16:14:47.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:47.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f7affd640 1 -- 172.21.15.83:0/3759860932 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f74033de0 con 0x7f8f840b32e0 2024-03-28T16:14:47.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:47.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f7affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:14:47.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f7affd640 10 monclient: dump: 2024-03-28T16:14:47.623 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:47.623 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:47.623 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.623 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.623 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:47.623 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:47.623 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:47.623 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:47.623 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:47.623 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:47.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f88d67640 4 set_mon_vals no callback set 2024-03-28T16:14:47.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f8c7ce640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:14:47.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f8c7ce640 10 monclient: _renew_subs 2024-03-28T16:14:47.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f8c7ce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:47.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f8c7ce640 1 -- 172.21.15.83:0/3759860932 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f8413c580 con 0x7f8f840b32e0 2024-03-28T16:14:47.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.619+0000 7f8f8c7ce640 1 librados: init done 2024-03-28T16:14:47.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.620+0000 7f8f8c7ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:47.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.620+0000 7f8f8c7ce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:47.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.620+0000 7f8f8c7ce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:47.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.620+0000 7f8f8c7ce640 1 -- 172.21.15.83:0/3759860932 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f841d9970 con 0x7f8f840b32e0 2024-03-28T16:14:47.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.623+0000 7f8f7affd640 1 -- 172.21.15.83:0/3759860932 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8f7403a040 con 0x7f8f840b32e0 2024-03-28T16:14:47.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.623+0000 7f8f7affd640 1 -- 172.21.15.83:0/3759860932 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f8f7403c820 con 0x7f8f840b32e0 2024-03-28T16:14:47.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.623+0000 7f8f7affd640 10 client.4129.objecter ms_dispatch 0x7f8f84000b90 osd_map(4..4 src has 1..4) v4 2024-03-28T16:14:47.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.623+0000 7f8f7affd640 3 client.4129.objecter handle_osd_map got epochs [4,4] > 0 2024-03-28T16:14:47.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.623+0000 7f8f7affd640 3 client.4129.objecter handle_osd_map decoding full epoch 4 2024-03-28T16:14:47.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.623+0000 7f8f7affd640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-28T16:14:47.716 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:47.716+0000 7f32d7241040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T16:14:47.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.721+0000 7f8f7affd640 1 -- 172.21.15.83:0/3759860932 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f8f74067b50 con 0x7f8f840b32e0 2024-03-28T16:14:47.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.721+0000 7f8f7affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:47.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.721+0000 7f8f7affd640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:14:47.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.792+0000 7f8f8c7ce640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0473d84c-e864-4373-b9d1-283130278363", "id": 2}] 2024-03-28T16:14:47.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.792+0000 7f8f8c7ce640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0473d84c-e864-4373-b9d1-283130278363", "id": 2}] 2024-03-28T16:14:47.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.792+0000 7f8f8c7ce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:47.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.792+0000 7f8f8c7ce640 1 -- 172.21.15.83:0/3759860932 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0473d84c-e864-4373-b9d1-283130278363", "id": 2} v 0) v1 -- 0x7f8f841d9d50 con 0x7f8f840b32e0 2024-03-28T16:14:47.794 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:47.794+0000 7f392f430040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T16:14:47.855 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:47.855+0000 7f32d7241040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T16:14:47.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.857+0000 7f8f7affd640 1 -- 172.21.15.83:0/3759860932 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0473d84c-e864-4373-b9d1-283130278363", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8f740560b0 con 0x7f8f840b32e0 2024-03-28T16:14:47.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.858+0000 7f8f7affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0473d84c-e864-4373-b9d1-283130278363", "id": 2}] 2024-03-28T16:14:47.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.858+0000 7f8f7affd640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:14:47.858 INFO:teuthology.orchestra.run.smithi083.stdout:2 2024-03-28T16:14:47.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.858+0000 7f8f8c7ce640 10 librados: watch_flush enter 2024-03-28T16:14:47.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.858+0000 7f8f8c7ce640 10 librados: watch_flush exit 2024-03-28T16:14:47.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.858+0000 7f8f8c7ce640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-28T16:14:47.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.858+0000 7f8f8c7ce640 10 client.4129.objecter successfully canceled tick 2024-03-28T16:14:47.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.858+0000 7f8f8c7ce640 10 monclient: shutdown 2024-03-28T16:14:47.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.858+0000 7f8f8c7ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:47.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.858+0000 7f8f8c7ce640 1 -- 172.21.15.83:0/3759860932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f840b32e0 msgr2=0x7f8f84140aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:47.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.858+0000 7f8f8c7ce640 1 --2- 172.21.15.83:0/3759860932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f840b32e0 0x7f8f84140aa0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8f7402b830 tx=0x7f8f740046c0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.859+0000 7f8f8c7ce640 1 -- 172.21.15.83:0/3759860932 shutdown_connections 2024-03-28T16:14:47.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.859+0000 7f8f8c7ce640 1 --2- 172.21.15.83:0/3759860932 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f8413ee50 0x7f8f841da030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.859+0000 7f8f8c7ce640 1 --2- 172.21.15.83:0/3759860932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f840b32e0 0x7f8f84140aa0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.859+0000 7f8f8c7ce640 1 --2- 172.21.15.83:0/3759860932 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f84071480 0x7f8f8413fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.859+0000 7f8f8c7ce640 1 -- 172.21.15.83:0/3759860932 >> 172.21.15.83:0/3759860932 conn(0x7f8f840b70e0 msgr2=0x7f8f840b4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:47.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.859+0000 7f8f8c7ce640 1 -- 172.21.15.83:0/3759860932 shutdown_connections 2024-03-28T16:14:47.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.859+0000 7f8f8c7ce640 1 -- 172.21.15.83:0/3759860932 wait complete. 2024-03-28T16:14:47.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.859+0000 7f8f8c7ce640 1 librados: shutdown 2024-03-28T16:14:47.869 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd new b64fd5e7-cdec-4bd0-b5e2-dfc8d4f6b9ce 3 2024-03-28T16:14:47.940 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:47.938+0000 7f392f430040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T16:14:47.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.968+0000 7fcf8fd37640 10 monclient: get_monmap_and_config 2024-03-28T16:14:47.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.968+0000 7fcf8fd37640 10 monclient: build_initial_monmap 2024-03-28T16:14:47.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.968+0000 7fcf8fd37640 10 monclient: monmap: 2024-03-28T16:14:47.968 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-28T16:14:47.968 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:47.968 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:14:47.968627+0000 2024-03-28T16:14:47.969 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:14:47.968627+0000 2024-03-28T16:14:47.969 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:47.969 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:47.969 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:14:47.969 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:14:47.969 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:14:47.969 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:14:47.969 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:14:47.969 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:47.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.968+0000 7fcf8fd37640 1 Processor -- start 2024-03-28T16:14:47.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.968+0000 7fcf8fd37640 1 -- start start 2024-03-28T16:14:47.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.968+0000 7fcf8fd37640 10 monclient: init 2024-03-28T16:14:47.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.969+0000 7fcf8fd37640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:47.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.969+0000 7fcf8fd37640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-03-28T16:14:47.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.969+0000 7fcf8fd37640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcf88071480 0x7fcf8813e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:47.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.969+0000 7fcf8fd37640 10 monclient(hunting): picked mon.noname-a con 0x7fcf88071480 addr v2:172.21.15.7:3300/0 2024-03-28T16:14:47.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.969+0000 7fcf8fd37640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcf880b32e0 addr v1:172.21.15.83:6789/0 2024-03-28T16:14:47.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.969+0000 7fcf8fd37640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcf8813ee50 0x7fcf88145110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:47.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.969+0000 7fcf8fd37640 10 monclient(hunting): picked mon.noname-b con 0x7fcf8813ee50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:47.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.969+0000 7fcf8fd37640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf880a8c90 con 0x7fcf880b32e0 2024-03-28T16:14:47.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.969+0000 7fcf8fd37640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:47.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.969+0000 7fcf8fd37640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf88071860 con 0x7fcf88071480 2024-03-28T16:14:47.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.969+0000 7fcf8fd37640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:47.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.969+0000 7fcf8fd37640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf88145650 con 0x7fcf8813ee50 2024-03-28T16:14:47.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.969+0000 7fcf8fd37640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:47.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.969+0000 7fcf8fd37640 10 monclient(hunting): authenticate will time out at 1355.138356s 2024-03-28T16:14:47.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.970+0000 7fcf8d2d2640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcf880b32e0 0x7fcf88144a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60890/0 (socket says 172.21.15.83:60890) 2024-03-28T16:14:47.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.970+0000 7fcf8d2d2640 1 -- 172.21.15.83:0/4172464181 learned_addr learned my addr 172.21.15.83:0/4172464181 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-28T16:14:47.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.970+0000 7fcf8dad3640 1 --2- 172.21.15.83:0/4172464181 >> v2:172.21.15.7:3300/0 conn(0x7fcf88071480 0x7fcf8813e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:47.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.970+0000 7fcf8e2d4640 1 --2- 172.21.15.83:0/4172464181 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcf8813ee50 0x7fcf88145110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:47.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.970+0000 7fcf8dad3640 10 monclient(hunting): get_auth_request con 0x7fcf88071480 auth_method 0 2024-03-28T16:14:47.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.970+0000 7fcf8dad3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:47.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.970+0000 7fcf8dad3640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:47.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.970+0000 7fcf8dad3640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:47.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.970+0000 7fcf8e2d4640 10 monclient(hunting): get_auth_request con 0x7fcf8813ee50 auth_method 0 2024-03-28T16:14:47.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.970+0000 7fcf8e2d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:47.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.970+0000 7fcf8e2d4640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:47.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.970+0000 7fcf8e2d4640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:47.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.971+0000 7fcf8dad3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:47.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.971+0000 7fcf8dad3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:47.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.971+0000 7fcf8dad3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:47.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.971+0000 7fcf8e2d4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:47.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.971+0000 7fcf8e2d4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:47.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.971+0000 7fcf8e2d4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:47.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.972+0000 7fcf8cad1640 1 -- 172.21.15.83:0/4172464181 <== mon.1 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057537656 0 0) 0x7fcf880a8c90 con 0x7fcf880b32e0 2024-03-28T16:14:47.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.972+0000 7fcf8cad1640 10 monclient(hunting): _negotiate 2024-03-28T16:14:47.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.972+0000 7fcf8cad1640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:47.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.972+0000 7fcf8cad1640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:47.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.972+0000 7fcf8cad1640 10 monclient(hunting): my global_id is 4132 2024-03-28T16:14:47.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.972+0000 7fcf8cad1640 1 -- 172.21.15.83:0/4172464181 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf74003520 con 0x7fcf880b32e0 2024-03-28T16:14:47.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.973+0000 7fcf8dad3640 10 monclient(hunting): handle_auth_done global_id 4131 payload 274 2024-03-28T16:14:47.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.973+0000 7fcf8dad3640 1 -- 172.21.15.83:0/4172464181 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcf8813ee50 msgr2=0x7fcf88145110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:14:47.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.973+0000 7fcf8dad3640 1 --2- 172.21.15.83:0/4172464181 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcf8813ee50 0x7fcf88145110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.973+0000 7fcf8dad3640 1 -- 172.21.15.83:0/4172464181 >> v1:172.21.15.83:6789/0 conn(0x7fcf880b32e0 legacy=0x7fcf88144a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:47.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.973+0000 7fcf8dad3640 10 monclient: _finish_hunting 0 2024-03-28T16:14:47.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.973+0000 7fcf8dad3640 1 monclient: found mon.noname-a 2024-03-28T16:14:47.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.973+0000 7fcf8dad3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:47.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.973+0000 7fcf8dad3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:14:47.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.973+0000 7fcf8dad3640 1 -- 172.21.15.83:0/4172464181 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf88072140 con 0x7fcf88071480 2024-03-28T16:14:47.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.976+0000 7fcf8dad3640 1 --2- 172.21.15.83:0/4172464181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88071480 0x7fcf8813e910 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcf840097a0 tx=0x7fcf8402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf39f9cc91a3c34 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:47.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.978+0000 7fcf8cad1640 1 -- 172.21.15.83:0/4172464181 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8402f050 con 0x7fcf88071480 2024-03-28T16:14:47.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.978+0000 7fcf8cad1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:47.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.978+0000 7fcf8cad1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:14:47.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.978+0000 7fcf8cad1640 10 monclient: dump: 2024-03-28T16:14:47.978 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:47.978 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:47.978 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.978 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.978 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:47.978 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:47.978 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:47.978 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:47.978 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:47.978 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:47.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.978+0000 7fcf8cad1640 10 monclient: _finish_auth 0 2024-03-28T16:14:47.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.978+0000 7fcf8cad1640 10 monclient: _check_auth_tickets 2024-03-28T16:14:47.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.978+0000 7fcf8cad1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:47.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.978+0000 7fcf8fd37640 5 monclient: authenticate success, global_id 4131 2024-03-28T16:14:47.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.978+0000 7fcf8fd37640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:14:47.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.979+0000 7fcf8cad1640 1 -- 172.21.15.83:0/4172464181 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf84031700 con 0x7fcf88071480 2024-03-28T16:14:47.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.979+0000 7fcf8cad1640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:47.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.979+0000 7fcf8cad1640 1 -- 172.21.15.83:0/4172464181 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf84031a70 con 0x7fcf88071480 2024-03-28T16:14:47.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.979+0000 7fcf8fd37640 10 monclient: get_monmap_and_config success 2024-03-28T16:14:47.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.979+0000 7fcf8fd37640 4 set_mon_vals no callback set 2024-03-28T16:14:47.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.979+0000 7fcf8cad1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:47.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.979+0000 7fcf8cad1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:14:47.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.979+0000 7fcf8cad1640 10 monclient: dump: 2024-03-28T16:14:47.980 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:47.980 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:47.980 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.980 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.980 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:47.980 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:47.980 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:47.980 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:47.980 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:47.980 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:47.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.980+0000 7fcf8fd37640 10 monclient: shutdown 2024-03-28T16:14:47.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.980+0000 7fcf8fd37640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:47.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.980+0000 7fcf8fd37640 1 -- 172.21.15.83:0/4172464181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88071480 msgr2=0x7fcf8813e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:47.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.980+0000 7fcf8fd37640 1 --2- 172.21.15.83:0/4172464181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88071480 0x7fcf8813e910 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcf840097a0 tx=0x7fcf8402b7a0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.980+0000 7fcf8fd37640 1 -- 172.21.15.83:0/4172464181 shutdown_connections 2024-03-28T16:14:47.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.980+0000 7fcf8fd37640 1 --2- 172.21.15.83:0/4172464181 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcf8813ee50 0x7fcf88145110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.980+0000 7fcf8fd37640 1 --2- 172.21.15.83:0/4172464181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88071480 0x7fcf8813e910 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.980+0000 7fcf8fd37640 1 -- 172.21.15.83:0/4172464181 >> 172.21.15.83:0/4172464181 conn(0x7fcf880b70e0 msgr2=0x7fcf880b74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:47.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.980+0000 7fcf8fd37640 1 -- 172.21.15.83:0/4172464181 shutdown_connections 2024-03-28T16:14:47.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.980+0000 7fcf8fd37640 1 -- 172.21.15.83:0/4172464181 wait complete. 2024-03-28T16:14:47.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.981+0000 7fcf8fd37640 10 monclient: build_initial_monmap 2024-03-28T16:14:47.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.981+0000 7fcf8fd37640 10 monclient: monmap: 2024-03-28T16:14:47.984 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-28T16:14:47.984 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:14:47.985 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-28T16:14:47.985 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-28T16:14:47.985 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:47.985 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:47.985 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:14:47.985 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:14:47.985 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:14:47.985 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:47.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.981+0000 7fcf8fd37640 1 librados: starting msgr at 2024-03-28T16:14:47.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.981+0000 7fcf8fd37640 1 librados: starting objecter 2024-03-28T16:14:47.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.981+0000 7fcf8fd37640 1 Processor -- start 2024-03-28T16:14:47.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.981+0000 7fcf8fd37640 1 -- start start 2024-03-28T16:14:47.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.981+0000 7fcf8fd37640 1 librados: setting wanted keys 2024-03-28T16:14:47.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.981+0000 7fcf8fd37640 1 librados: calling monclient init 2024-03-28T16:14:47.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.981+0000 7fcf8fd37640 10 monclient: init 2024-03-28T16:14:47.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.982+0000 7fcf8fd37640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:47.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.982+0000 7fcf8fd37640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-28T16:14:47.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.982+0000 7fcf8fd37640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88071480 0x7fcf8813dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:47.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.982+0000 7fcf8fd37640 10 monclient(hunting): picked mon.noname-a con 0x7fcf88071480 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:14:47.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.982+0000 7fcf8fd37640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcf880b32e0 0x7fcf8813c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:47.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.982+0000 7fcf8fd37640 10 monclient(hunting): picked mon.noname-c con 0x7fcf880b32e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:47.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.982+0000 7fcf8fd37640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf8813ee50 0x7fcf88143600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:47.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.982+0000 7fcf8fd37640 10 monclient(hunting): picked mon.noname-b con 0x7fcf8813ee50 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:14:47.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.982+0000 7fcf8fd37640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:47.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.982+0000 7fcf8fd37640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf88144580 con 0x7fcf88071480 2024-03-28T16:14:47.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.982+0000 7fcf8fd37640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:47.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.982+0000 7fcf8fd37640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf881429a0 con 0x7fcf8813ee50 2024-03-28T16:14:47.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.982+0000 7fcf8fd37640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:47.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.982+0000 7fcf8fd37640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf881440a0 con 0x7fcf880b32e0 2024-03-28T16:14:47.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.982+0000 7fcf8fd37640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:47.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.982+0000 7fcf8fd37640 10 monclient(hunting): authenticate will time out at 1355.151267s 2024-03-28T16:14:47.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.982+0000 7fcf8e2d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf8813ee50 0x7fcf88143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:47.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.982+0000 7fcf8dad3640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88071480 0x7fcf8813dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:47.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.983+0000 7fcf8d2d2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcf880b32e0 0x7fcf8813c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:47.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.983+0000 7fcf8e2d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf8813ee50 0x7fcf88143600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:35996/0 (socket says 172.21.15.83:35996) 2024-03-28T16:14:47.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.983+0000 7fcf8e2d4640 1 -- 172.21.15.83:0/2691368085 learned_addr learned my addr 172.21.15.83:0/2691368085 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-28T16:14:47.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.983+0000 7fcf8e2d4640 10 monclient(hunting): get_auth_request con 0x7fcf8813ee50 auth_method 0 2024-03-28T16:14:47.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.983+0000 7fcf8e2d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:47.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.983+0000 7fcf8e2d4640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:47.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.983+0000 7fcf8e2d4640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:47.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.983+0000 7fcf8dad3640 10 monclient(hunting): get_auth_request con 0x7fcf88071480 auth_method 0 2024-03-28T16:14:47.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.983+0000 7fcf8dad3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:47.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.983+0000 7fcf8dad3640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:47.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.983+0000 7fcf8dad3640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:47.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.983+0000 7fcf8d2d2640 10 monclient(hunting): get_auth_request con 0x7fcf880b32e0 auth_method 0 2024-03-28T16:14:47.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.983+0000 7fcf8d2d2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:47.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.983+0000 7fcf8d2d2640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:47.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.983+0000 7fcf8d2d2640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:47.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.983+0000 7fcf8e2d4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:47.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.983+0000 7fcf8e2d4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:47.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.983+0000 7fcf8e2d4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:47.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.983+0000 7fcf8dad3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:47.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.983+0000 7fcf8dad3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:47.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.983+0000 7fcf8dad3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:47.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.984+0000 7fcf8d2d2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:47.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.984+0000 7fcf8d2d2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:47.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.984+0000 7fcf8d2d2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:47.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.986+0000 7fcf8e2d4640 10 monclient(hunting): handle_auth_done global_id 4135 payload 824 2024-03-28T16:14:47.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.986+0000 7fcf8e2d4640 1 -- 172.21.15.83:0/2691368085 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcf880b32e0 msgr2=0x7fcf8813c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:47.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.986+0000 7fcf8e2d4640 1 --2- 172.21.15.83:0/2691368085 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcf880b32e0 0x7fcf8813c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.986+0000 7fcf8e2d4640 1 -- 172.21.15.83:0/2691368085 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88071480 msgr2=0x7fcf8813dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:47.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.986+0000 7fcf8e2d4640 1 --2- 172.21.15.83:0/2691368085 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88071480 0x7fcf8813dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:47.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.986+0000 7fcf8e2d4640 10 monclient: _finish_hunting 0 2024-03-28T16:14:47.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.986+0000 7fcf8e2d4640 1 monclient: found mon.noname-b 2024-03-28T16:14:47.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.986+0000 7fcf8e2d4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:47.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.986+0000 7fcf8e2d4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:14:47.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.986+0000 7fcf8e2d4640 1 -- 172.21.15.83:0/2691368085 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf84009400 con 0x7fcf8813ee50 2024-03-28T16:14:47.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.988+0000 7fcf8e2d4640 1 --2- 172.21.15.83:0/2691368085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf8813ee50 0x7fcf88143600 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcf80009cb0 tx=0x7fcf8000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:47.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.988+0000 7fcf7e7fc640 10 client.?.objecter ms_handle_connect 0x7fcf8813ee50 2024-03-28T16:14:47.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.988+0000 7fcf7e7fc640 10 client.?.objecter resend_mon_ops 2024-03-28T16:14:47.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf7e7fc640 1 -- 172.21.15.83:0/2691368085 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf80009b20 con 0x7fcf8813ee50 2024-03-28T16:14:47.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:47.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf7e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:14:47.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf7e7fc640 10 monclient: dump: 2024-03-28T16:14:47.994 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:47.994 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:47.994 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.995 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.995 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:47.995 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:47.995 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:47.995 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:47.995 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:47.995 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:47.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf7e7fc640 10 monclient: _finish_auth 0 2024-03-28T16:14:47.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf7e7fc640 10 monclient: _check_auth_tickets 2024-03-28T16:14:47.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:47.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf7e7fc640 1 -- 172.21.15.83:0/2691368085 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8000f040 con 0x7fcf8813ee50 2024-03-28T16:14:47.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf8fd37640 5 monclient: authenticate success, global_id 4135 2024-03-28T16:14:47.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf8fd37640 10 monclient: _renew_subs 2024-03-28T16:14:47.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf8fd37640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:47.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf8fd37640 1 -- 172.21.15.83:0/2691368085 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf881301b0 con 0x7fcf8813ee50 2024-03-28T16:14:47.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:47.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf7e7fc640 1 -- 172.21.15.83:0/2691368085 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf800146c0 con 0x7fcf8813ee50 2024-03-28T16:14:47.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:47.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf7e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:14:47.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf7e7fc640 10 monclient: dump: 2024-03-28T16:14:47.996 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:47.996 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:47.996 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.996 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:47.996 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:47.996 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:47.996 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:47.996 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:47.996 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:47.996 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:47.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf7ffff640 4 set_mon_vals no callback set 2024-03-28T16:14:47.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf8fd37640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:14:47.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf8fd37640 10 monclient: _renew_subs 2024-03-28T16:14:47.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf8fd37640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:47.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf8fd37640 1 -- 172.21.15.83:0/2691368085 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf88143d20 con 0x7fcf8813ee50 2024-03-28T16:14:47.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.991+0000 7fcf8fd37640 1 librados: init done 2024-03-28T16:14:47.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.992+0000 7fcf8fd37640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:47.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.992+0000 7fcf8fd37640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:47.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.992+0000 7fcf8fd37640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:47.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.992+0000 7fcf8fd37640 1 -- 172.21.15.83:0/2691368085 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf881446c0 con 0x7fcf8813ee50 2024-03-28T16:14:47.997 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:47.993+0000 7f32d7241040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T16:14:47.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.994+0000 7fcf7e7fc640 1 -- 172.21.15.83:0/2691368085 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fcf80014b20 con 0x7fcf8813ee50 2024-03-28T16:14:47.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.994+0000 7fcf7e7fc640 1 -- 172.21.15.83:0/2691368085 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fcf8001dba0 con 0x7fcf8813ee50 2024-03-28T16:14:47.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.994+0000 7fcf7e7fc640 10 client.4135.objecter ms_dispatch 0x7fcf88000b90 osd_map(5..5 src has 1..5) v4 2024-03-28T16:14:47.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.994+0000 7fcf7e7fc640 3 client.4135.objecter handle_osd_map got epochs [5,5] > 0 2024-03-28T16:14:47.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.994+0000 7fcf7e7fc640 3 client.4135.objecter handle_osd_map decoding full epoch 5 2024-03-28T16:14:47.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:47.994+0000 7fcf7e7fc640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-28T16:14:48.040 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:48.040+0000 7f32d7241040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T16:14:48.071 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:48.070+0000 7f392f430040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T16:14:48.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.090+0000 7fcf7e7fc640 1 -- 172.21.15.83:0/2691368085 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fcf8003bb50 con 0x7fcf8813ee50 2024-03-28T16:14:48.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.090+0000 7fcf7e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:48.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.090+0000 7fcf7e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:14:48.118 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:48.117+0000 7f392f430040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T16:14:48.153 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:48.153+0000 7f32d7241040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T16:14:48.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.160+0000 7fcf8fd37640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b64fd5e7-cdec-4bd0-b5e2-dfc8d4f6b9ce", "id": 3}] 2024-03-28T16:14:48.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.160+0000 7fcf8fd37640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b64fd5e7-cdec-4bd0-b5e2-dfc8d4f6b9ce", "id": 3}] 2024-03-28T16:14:48.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.160+0000 7fcf8fd37640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:48.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.160+0000 7fcf8fd37640 1 -- 172.21.15.83:0/2691368085 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b64fd5e7-cdec-4bd0-b5e2-dfc8d4f6b9ce", "id": 3} v 0) v1 -- 0x7fcf880b2270 con 0x7fcf8813ee50 2024-03-28T16:14:48.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.236+0000 7fcf7e7fc640 1 -- 172.21.15.83:0/2691368085 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b64fd5e7-cdec-4bd0-b5e2-dfc8d4f6b9ce", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcf8002a070 con 0x7fcf8813ee50 2024-03-28T16:14:48.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.236+0000 7fcf7e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b64fd5e7-cdec-4bd0-b5e2-dfc8d4f6b9ce", "id": 3}] 2024-03-28T16:14:48.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.236+0000 7fcf7e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:14:48.237 INFO:teuthology.orchestra.run.smithi083.stdout:3 2024-03-28T16:14:48.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.237+0000 7fcf8fd37640 10 librados: watch_flush enter 2024-03-28T16:14:48.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.237+0000 7fcf8fd37640 10 librados: watch_flush exit 2024-03-28T16:14:48.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.237+0000 7fcf8fd37640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-28T16:14:48.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.237+0000 7fcf8fd37640 10 client.4135.objecter successfully canceled tick 2024-03-28T16:14:48.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.237+0000 7fcf8fd37640 10 monclient: shutdown 2024-03-28T16:14:48.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.237+0000 7fcf8fd37640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:48.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.237+0000 7fcf8fd37640 1 -- 172.21.15.83:0/2691368085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf8813ee50 msgr2=0x7fcf88143600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:48.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.237+0000 7fcf8fd37640 1 --2- 172.21.15.83:0/2691368085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf8813ee50 0x7fcf88143600 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcf80009cb0 tx=0x7fcf8000c490 comp rx=0 tx=0).stop 2024-03-28T16:14:48.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.237+0000 7fcf8fd37640 1 -- 172.21.15.83:0/2691368085 shutdown_connections 2024-03-28T16:14:48.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.237+0000 7fcf8fd37640 1 --2- 172.21.15.83:0/2691368085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf8813ee50 0x7fcf88143600 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:48.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.237+0000 7fcf8fd37640 1 --2- 172.21.15.83:0/2691368085 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcf880b32e0 0x7fcf8813c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:48.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.237+0000 7fcf8fd37640 1 --2- 172.21.15.83:0/2691368085 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88071480 0x7fcf8813dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:48.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.237+0000 7fcf8fd37640 1 -- 172.21.15.83:0/2691368085 >> 172.21.15.83:0/2691368085 conn(0x7fcf880b70e0 msgr2=0x7fcf880b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:48.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.237+0000 7fcf8fd37640 1 -- 172.21.15.83:0/2691368085 shutdown_connections 2024-03-28T16:14:48.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.237+0000 7fcf8fd37640 1 -- 172.21.15.83:0/2691368085 wait complete. 2024-03-28T16:14:48.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.237+0000 7fcf8fd37640 1 librados: shutdown 2024-03-28T16:14:48.239 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:48.238+0000 7f392f430040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T16:14:48.247 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd new 8f7552db-5651-430f-a826-3c6843bffb99 4 2024-03-28T16:14:48.252 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:48.252+0000 7f32d7241040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T16:14:48.341 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:48.340+0000 7f392f430040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T16:14:48.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.346+0000 7f13847f0640 10 monclient: get_monmap_and_config 2024-03-28T16:14:48.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.346+0000 7f13847f0640 10 monclient: build_initial_monmap 2024-03-28T16:14:48.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.346+0000 7f13847f0640 10 monclient: monmap: 2024-03-28T16:14:48.347 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-28T16:14:48.347 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:48.347 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:14:48.347202+0000 2024-03-28T16:14:48.347 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:14:48.347202+0000 2024-03-28T16:14:48.347 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:48.347 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:48.347 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:14:48.347 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:14:48.347 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:14:48.347 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:14:48.347 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:14:48.348 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:48.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.347+0000 7f13847f0640 1 Processor -- start 2024-03-28T16:14:48.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.347+0000 7f13847f0640 1 -- start start 2024-03-28T16:14:48.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.347+0000 7f13847f0640 10 monclient: init 2024-03-28T16:14:48.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f13847f0640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:48.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f13847f0640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-03-28T16:14:48.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f13847f0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f137c071480 0x7f137c13e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:48.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f13847f0640 10 monclient(hunting): picked mon.noname-b con 0x7f137c071480 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:48.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f13847f0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f137c0b32e0 addr v1:172.21.15.83:6789/0 2024-03-28T16:14:48.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f13847f0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f137c13ee50 0x7f137c145110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:48.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f13847f0640 10 monclient(hunting): picked mon.noname-c con 0x7f137c13ee50 addr v2:172.21.15.83:3300/0 2024-03-28T16:14:48.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f13847f0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f137c0a8c90 con 0x7f137c0b32e0 2024-03-28T16:14:48.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f13847f0640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:48.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f13847f0640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f137c071860 con 0x7f137c13ee50 2024-03-28T16:14:48.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f13847f0640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:48.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f13847f0640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f137c145650 con 0x7f137c071480 2024-03-28T16:14:48.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f13847f0640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:48.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f13847f0640 10 monclient(hunting): authenticate will time out at 1355.516958s 2024-03-28T16:14:48.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f1382d8d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f137c13ee50 0x7f137c145110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:48.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f138258c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f137c071480 0x7f137c13e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:48.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f138258c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f137c071480 0x7f137c13e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.83:42586/0 (socket says 172.21.15.83:42586) 2024-03-28T16:14:48.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f138258c640 1 -- 172.21.15.83:0/2053106952 learned_addr learned my addr 172.21.15.83:0/2053106952 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-28T16:14:48.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f138258c640 10 monclient(hunting): get_auth_request con 0x7f137c071480 auth_method 0 2024-03-28T16:14:48.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f138258c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:48.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f138258c640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:48.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.348+0000 7f138258c640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:48.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.349+0000 7f1382d8d640 10 monclient(hunting): get_auth_request con 0x7f137c13ee50 auth_method 0 2024-03-28T16:14:48.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.349+0000 7f1382d8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:48.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.349+0000 7f1382d8d640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:48.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.349+0000 7f1382d8d640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:48.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.349+0000 7f138258c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:48.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.349+0000 7f138258c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:48.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.349+0000 7f138258c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:48.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.350+0000 7f1382d8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:48.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.350+0000 7f1382d8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:48.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.350+0000 7f1382d8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:48.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.351+0000 7f138258c640 10 monclient(hunting): handle_auth_done global_id 4139 payload 274 2024-03-28T16:14:48.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.351+0000 7f138258c640 1 -- 172.21.15.83:0/2053106952 >> v2:172.21.15.83:3300/0 conn(0x7f137c13ee50 msgr2=0x7f137c145110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:14:48.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.351+0000 7f138258c640 1 --2- 172.21.15.83:0/2053106952 >> v2:172.21.15.83:3300/0 conn(0x7f137c13ee50 0x7f137c145110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:48.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.351+0000 7f138258c640 1 -- 172.21.15.83:0/2053106952 >> v1:172.21.15.83:6789/0 conn(0x7f137c0b32e0 legacy=0x7f137c144a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:48.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.351+0000 7f138258c640 10 monclient: _finish_hunting 0 2024-03-28T16:14:48.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.351+0000 7f138258c640 1 monclient: found mon.noname-b 2024-03-28T16:14:48.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.351+0000 7f138258c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:48.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.351+0000 7f138258c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-03-28T16:14:48.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.351+0000 7f138258c640 1 -- 172.21.15.83:0/2053106952 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f137c072140 con 0x7f137c071480 2024-03-28T16:14:48.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.353+0000 7f138258c640 1 --2- 172.21.15.83:0/2053106952 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f137c071480 0x7f137c13e910 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1378009870 tx=0x7f137802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b2c5f494f97ee56 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:48.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.355+0000 7f138158a640 1 -- 172.21.15.83:0/2053106952 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13780317c0 con 0x7f137c071480 2024-03-28T16:14:48.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.355+0000 7f138158a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:48.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.355+0000 7f138158a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:14:48.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.355+0000 7f138158a640 10 monclient: dump: 2024-03-28T16:14:48.356 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:48.356 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:48.356 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:48.356 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:48.356 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:48.357 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:48.357 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:48.357 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:48.357 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:48.357 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:48.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.355+0000 7f138158a640 10 monclient: _finish_auth 0 2024-03-28T16:14:48.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.355+0000 7f138158a640 10 monclient: _check_auth_tickets 2024-03-28T16:14:48.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.355+0000 7f138158a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:48.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.355+0000 7f13847f0640 5 monclient: authenticate success, global_id 4139 2024-03-28T16:14:48.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.355+0000 7f13847f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:14:48.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.357+0000 7f138158a640 1 -- 172.21.15.83:0/2053106952 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1378031920 con 0x7f137c071480 2024-03-28T16:14:48.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.357+0000 7f138158a640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:48.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.357+0000 7f138158a640 1 -- 172.21.15.83:0/2053106952 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1378031cb0 con 0x7f137c071480 2024-03-28T16:14:48.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.357+0000 7f138158a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:48.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.357+0000 7f138158a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:14:48.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.357+0000 7f138158a640 10 monclient: dump: 2024-03-28T16:14:48.359 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:48.359 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:48.359 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:48.359 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:48.359 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:48.359 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:48.359 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:48.359 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:48.359 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:48.359 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:48.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.357+0000 7f13847f0640 10 monclient: get_monmap_and_config success 2024-03-28T16:14:48.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.357+0000 7f13847f0640 4 set_mon_vals no callback set 2024-03-28T16:14:48.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.358+0000 7f13847f0640 10 monclient: shutdown 2024-03-28T16:14:48.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.358+0000 7f13847f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:48.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.358+0000 7f13847f0640 1 -- 172.21.15.83:0/2053106952 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f137c071480 msgr2=0x7f137c13e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:48.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.358+0000 7f13847f0640 1 --2- 172.21.15.83:0/2053106952 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f137c071480 0x7f137c13e910 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1378009870 tx=0x7f137802b860 comp rx=0 tx=0).stop 2024-03-28T16:14:48.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.358+0000 7f13847f0640 1 -- 172.21.15.83:0/2053106952 shutdown_connections 2024-03-28T16:14:48.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.358+0000 7f13847f0640 1 --2- 172.21.15.83:0/2053106952 >> v2:172.21.15.83:3300/0 conn(0x7f137c13ee50 0x7f137c145110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:48.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.358+0000 7f13847f0640 1 --2- 172.21.15.83:0/2053106952 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f137c071480 0x7f137c13e910 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:48.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.358+0000 7f13847f0640 1 -- 172.21.15.83:0/2053106952 >> 172.21.15.83:0/2053106952 conn(0x7f137c0b70e0 msgr2=0x7f137c0b74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:48.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.358+0000 7f13847f0640 1 -- 172.21.15.83:0/2053106952 shutdown_connections 2024-03-28T16:14:48.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.358+0000 7f13847f0640 1 -- 172.21.15.83:0/2053106952 wait complete. 2024-03-28T16:14:48.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.359+0000 7f13847f0640 10 monclient: build_initial_monmap 2024-03-28T16:14:48.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.359+0000 7f13847f0640 10 monclient: monmap: 2024-03-28T16:14:48.362 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-28T16:14:48.362 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:14:48.362 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-28T16:14:48.362 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-28T16:14:48.363 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:48.363 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:48.363 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:14:48.363 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:14:48.363 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:14:48.363 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:48.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.359+0000 7f13847f0640 1 librados: starting msgr at 2024-03-28T16:14:48.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.359+0000 7f13847f0640 1 librados: starting objecter 2024-03-28T16:14:48.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.359+0000 7f13847f0640 1 Processor -- start 2024-03-28T16:14:48.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.359+0000 7f13847f0640 1 -- start start 2024-03-28T16:14:48.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.359+0000 7f13847f0640 1 librados: setting wanted keys 2024-03-28T16:14:48.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.359+0000 7f13847f0640 1 librados: calling monclient init 2024-03-28T16:14:48.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.359+0000 7f13847f0640 10 monclient: init 2024-03-28T16:14:48.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.360+0000 7f13847f0640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:48.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.360+0000 7f13847f0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-28T16:14:48.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.360+0000 7f13847f0640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f137c071480 0x7f137c143d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:48.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.360+0000 7f13847f0640 10 monclient(hunting): picked mon.noname-a con 0x7f137c071480 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:14:48.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.360+0000 7f13847f0640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f137c0b32e0 0x7f137c1346e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:48.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.360+0000 7f13847f0640 10 monclient(hunting): picked mon.noname-c con 0x7f137c0b32e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:48.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.360+0000 7f13847f0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f137c13ee50 0x7f137c1d6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:48.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.360+0000 7f13847f0640 10 monclient(hunting): picked mon.noname-b con 0x7f137c13ee50 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:14:48.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.360+0000 7f13847f0640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:48.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.360+0000 7f13847f0640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137c142900 con 0x7f137c071480 2024-03-28T16:14:48.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.360+0000 7f13847f0640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:48.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.360+0000 7f13847f0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137c13de00 con 0x7f137c13ee50 2024-03-28T16:14:48.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.360+0000 7f13847f0640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:48.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.360+0000 7f13847f0640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f137c13e320 con 0x7f137c0b32e0 2024-03-28T16:14:48.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.360+0000 7f13847f0640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:48.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.360+0000 7f13847f0640 10 monclient(hunting): authenticate will time out at 1355.529675s 2024-03-28T16:14:48.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.361+0000 7f1382d8d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f137c13ee50 0x7f137c1d6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:48.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.361+0000 7f1382d8d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f137c13ee50 0x7f137c1d6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:36014/0 (socket says 172.21.15.83:36014) 2024-03-28T16:14:48.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.361+0000 7f1382d8d640 1 -- 172.21.15.83:0/1406954150 learned_addr learned my addr 172.21.15.83:0/1406954150 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-28T16:14:48.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.361+0000 7f1382d8d640 10 monclient(hunting): get_auth_request con 0x7f137c13ee50 auth_method 0 2024-03-28T16:14:48.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.361+0000 7f1382d8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:48.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.361+0000 7f1382d8d640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:48.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.361+0000 7f1382d8d640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:48.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.362+0000 7f1382d8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:48.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.362+0000 7f1382d8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:48.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.362+0000 7f1382d8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:48.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.362+0000 7f138258c640 1 --2- 172.21.15.83:0/1406954150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f137c071480 0x7f137c143d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:48.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.363+0000 7f138258c640 10 monclient(hunting): get_auth_request con 0x7f137c071480 auth_method 0 2024-03-28T16:14:48.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.363+0000 7f138258c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:48.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.363+0000 7f138258c640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:48.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.363+0000 7f138258c640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:48.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.363+0000 7f1381d8b640 1 --2- 172.21.15.83:0/1406954150 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f137c0b32e0 0x7f137c1346e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:48.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.363+0000 7f138258c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:48.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.363+0000 7f138258c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:48.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.363+0000 7f138258c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:48.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.363+0000 7f1381d8b640 10 monclient(hunting): get_auth_request con 0x7f137c0b32e0 auth_method 0 2024-03-28T16:14:48.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.363+0000 7f1381d8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:48.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.363+0000 7f1381d8b640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:48.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.363+0000 7f1381d8b640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:48.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.364+0000 7f1381d8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:48.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.364+0000 7f1381d8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:48.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.364+0000 7f1381d8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:48.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.365+0000 7f1382d8d640 10 monclient(hunting): handle_auth_done global_id 4144 payload 824 2024-03-28T16:14:48.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.365+0000 7f1382d8d640 1 -- 172.21.15.83:0/1406954150 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f137c0b32e0 msgr2=0x7f137c1346e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:48.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.365+0000 7f1382d8d640 1 --2- 172.21.15.83:0/1406954150 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f137c0b32e0 0x7f137c1346e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:48.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.365+0000 7f1382d8d640 1 -- 172.21.15.83:0/1406954150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f137c071480 msgr2=0x7f137c143d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:48.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.365+0000 7f1382d8d640 1 --2- 172.21.15.83:0/1406954150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f137c071480 0x7f137c143d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:48.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.365+0000 7f1382d8d640 10 monclient: _finish_hunting 0 2024-03-28T16:14:48.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.365+0000 7f1382d8d640 1 monclient: found mon.noname-b 2024-03-28T16:14:48.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.365+0000 7f1382d8d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:48.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.365+0000 7f1382d8d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:14:48.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.365+0000 7f1382d8d640 1 -- 172.21.15.83:0/1406954150 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13780094d0 con 0x7f137c13ee50 2024-03-28T16:14:48.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.367+0000 7f1382d8d640 1 --2- 172.21.15.83:0/1406954150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f137c13ee50 0x7f137c1d6720 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f137400efc0 tx=0x7f137400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:48.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.367+0000 7f1372ffd640 10 client.?.objecter ms_handle_connect 0x7f137c13ee50 2024-03-28T16:14:48.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.367+0000 7f1372ffd640 10 client.?.objecter resend_mon_ops 2024-03-28T16:14:48.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f1372ffd640 1 -- 172.21.15.83:0/1406954150 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1374009b20 con 0x7f137c13ee50 2024-03-28T16:14:48.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f1372ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:48.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f1372ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:14:48.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f1372ffd640 10 monclient: dump: 2024-03-28T16:14:48.371 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:48.371 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:48.371 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:48.371 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:48.371 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:48.371 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:48.371 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:48.371 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:48.372 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:48.372 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:48.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f1372ffd640 10 monclient: _finish_auth 0 2024-03-28T16:14:48.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f1372ffd640 10 monclient: _check_auth_tickets 2024-03-28T16:14:48.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f1372ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:48.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f1372ffd640 1 -- 172.21.15.83:0/1406954150 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f137400f040 con 0x7f137c13ee50 2024-03-28T16:14:48.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f13847f0640 5 monclient: authenticate success, global_id 4144 2024-03-28T16:14:48.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f13847f0640 10 monclient: _renew_subs 2024-03-28T16:14:48.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f13847f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:48.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f13847f0640 1 -- 172.21.15.83:0/1406954150 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f137c13dab0 con 0x7f137c13ee50 2024-03-28T16:14:48.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f1372ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:48.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f1372ffd640 1 -- 172.21.15.83:0/1406954150 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13740146c0 con 0x7f137c13ee50 2024-03-28T16:14:48.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f1372ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:48.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f1372ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:14:48.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f1372ffd640 10 monclient: dump: 2024-03-28T16:14:48.372 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:48.373 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:48.373 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:48.373 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:48.373 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:48.373 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:48.373 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:48.373 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:48.373 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:48.373 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:48.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f1380d89640 4 set_mon_vals no callback set 2024-03-28T16:14:48.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f13847f0640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:14:48.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f13847f0640 10 monclient: _renew_subs 2024-03-28T16:14:48.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f13847f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:48.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f13847f0640 1 -- 172.21.15.83:0/1406954150 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f137c134e00 con 0x7f137c13ee50 2024-03-28T16:14:48.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.370+0000 7f13847f0640 1 librados: init done 2024-03-28T16:14:48.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.371+0000 7f13847f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:48.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.371+0000 7f13847f0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:48.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.371+0000 7f13847f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:48.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.371+0000 7f13847f0640 1 -- 172.21.15.83:0/1406954150 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f137c13dfc0 con 0x7f137c13ee50 2024-03-28T16:14:48.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.374+0000 7f1372ffd640 1 -- 172.21.15.83:0/1406954150 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1374014b20 con 0x7f137c13ee50 2024-03-28T16:14:48.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.374+0000 7f1372ffd640 1 -- 172.21.15.83:0/1406954150 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f137401dbb0 con 0x7f137c13ee50 2024-03-28T16:14:48.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.374+0000 7f1372ffd640 10 client.4144.objecter ms_dispatch 0x7f137c000b90 osd_map(6..6 src has 1..6) v4 2024-03-28T16:14:48.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.374+0000 7f1372ffd640 3 client.4144.objecter handle_osd_map got epochs [6,6] > 0 2024-03-28T16:14:48.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.374+0000 7f1372ffd640 3 client.4144.objecter handle_osd_map decoding full epoch 6 2024-03-28T16:14:48.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.374+0000 7f1372ffd640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-28T16:14:48.415 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:48.415+0000 7f32d7241040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T16:14:48.459 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:48.458+0000 7f32d7241040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T16:14:48.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.471+0000 7f1372ffd640 1 -- 172.21.15.83:0/1406954150 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f137403bb50 con 0x7f137c13ee50 2024-03-28T16:14:48.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.471+0000 7f1372ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:48.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.471+0000 7f1372ffd640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:14:48.508 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:48.507+0000 7f392f430040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T16:14:48.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.545+0000 7f13847f0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8f7552db-5651-430f-a826-3c6843bffb99", "id": 4}] 2024-03-28T16:14:48.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.545+0000 7f13847f0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8f7552db-5651-430f-a826-3c6843bffb99", "id": 4}] 2024-03-28T16:14:48.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.545+0000 7f13847f0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:48.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.545+0000 7f13847f0640 1 -- 172.21.15.83:0/1406954150 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8f7552db-5651-430f-a826-3c6843bffb99", "id": 4} v 0) v1 -- 0x7f137c13a150 con 0x7f137c13ee50 2024-03-28T16:14:48.552 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:48.551+0000 7f392f430040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T16:14:48.619 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:48.618+0000 7f32d7241040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T16:14:48.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.629+0000 7f1372ffd640 1 -- 172.21.15.83:0/1406954150 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8f7552db-5651-430f-a826-3c6843bffb99", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f137402a070 con 0x7f137c13ee50 2024-03-28T16:14:48.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.629+0000 7f1372ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8f7552db-5651-430f-a826-3c6843bffb99", "id": 4}] 2024-03-28T16:14:48.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.629+0000 7f1372ffd640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:14:48.631 INFO:teuthology.orchestra.run.smithi083.stdout:4 2024-03-28T16:14:48.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.630+0000 7f13847f0640 10 librados: watch_flush enter 2024-03-28T16:14:48.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.630+0000 7f13847f0640 10 librados: watch_flush exit 2024-03-28T16:14:48.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.630+0000 7f13847f0640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-28T16:14:48.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.630+0000 7f13847f0640 10 client.4144.objecter successfully canceled tick 2024-03-28T16:14:48.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.630+0000 7f13847f0640 10 monclient: shutdown 2024-03-28T16:14:48.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.630+0000 7f13847f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:48.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.630+0000 7f13847f0640 1 -- 172.21.15.83:0/1406954150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f137c13ee50 msgr2=0x7f137c1d6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:48.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.630+0000 7f13847f0640 1 --2- 172.21.15.83:0/1406954150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f137c13ee50 0x7f137c1d6720 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f137400efc0 tx=0x7f137400c490 comp rx=0 tx=0).stop 2024-03-28T16:14:48.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.630+0000 7f13847f0640 1 -- 172.21.15.83:0/1406954150 shutdown_connections 2024-03-28T16:14:48.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.630+0000 7f13847f0640 1 --2- 172.21.15.83:0/1406954150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f137c13ee50 0x7f137c1d6720 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:48.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.630+0000 7f13847f0640 1 --2- 172.21.15.83:0/1406954150 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f137c0b32e0 0x7f137c1346e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:48.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.630+0000 7f13847f0640 1 --2- 172.21.15.83:0/1406954150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f137c071480 0x7f137c143d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:48.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.630+0000 7f13847f0640 1 -- 172.21.15.83:0/1406954150 >> 172.21.15.83:0/1406954150 conn(0x7f137c0b70e0 msgr2=0x7f137c0b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:48.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.630+0000 7f13847f0640 1 -- 172.21.15.83:0/1406954150 shutdown_connections 2024-03-28T16:14:48.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.630+0000 7f13847f0640 1 -- 172.21.15.83:0/1406954150 wait complete. 2024-03-28T16:14:48.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.630+0000 7f13847f0640 1 librados: shutdown 2024-03-28T16:14:48.641 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd new 2aabdbd2-d870-4d6d-823e-89b008ea3108 5 2024-03-28T16:14:48.718 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:48.717+0000 7f392f430040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T16:14:48.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.781+0000 7f4c47900640 10 monclient: get_monmap_and_config 2024-03-28T16:14:48.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.781+0000 7f4c47900640 10 monclient: build_initial_monmap 2024-03-28T16:14:48.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.781+0000 7f4c47900640 10 monclient: monmap: 2024-03-28T16:14:48.782 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-28T16:14:48.782 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:48.782 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:14:48.782647+0000 2024-03-28T16:14:48.782 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:14:48.782647+0000 2024-03-28T16:14:48.782 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:48.782 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:48.782 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:14:48.782 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:14:48.783 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:14:48.783 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:14:48.783 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:14:48.783 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:48.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.781+0000 7f4c47900640 1 Processor -- start 2024-03-28T16:14:48.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.781+0000 7f4c47900640 1 -- start start 2024-03-28T16:14:48.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.781+0000 7f4c47900640 10 monclient: init 2024-03-28T16:14:48.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.782+0000 7f4c47900640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:48.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.782+0000 7f4c47900640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-03-28T16:14:48.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.782+0000 7f4c47900640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c40071920 0x7f4c4013e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:48.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.782+0000 7f4c47900640 10 monclient(hunting): picked mon.noname-b con 0x7f4c40071920 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:48.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.782+0000 7f4c47900640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4c400a89a0 0x7f4c4013ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:48.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.782+0000 7f4c47900640 10 monclient(hunting): picked mon.noname-a con 0x7f4c400a89a0 addr v2:172.21.15.7:3300/0 2024-03-28T16:14:48.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.782+0000 7f4c47900640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4c400b1890 addr v1:172.21.15.7:6789/0 2024-03-28T16:14:48.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.782+0000 7f4c47900640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c400a8ec0 con 0x7f4c400b1890 2024-03-28T16:14:48.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.782+0000 7f4c47900640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:48.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.782+0000 7f4c47900640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c400aa480 con 0x7f4c400a89a0 2024-03-28T16:14:48.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.782+0000 7f4c47900640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:48.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.782+0000 7f4c47900640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c400b2430 con 0x7f4c40071920 2024-03-28T16:14:48.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.782+0000 7f4c47900640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:48.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.782+0000 7f4c47900640 10 monclient(hunting): authenticate will time out at 1355.952388s 2024-03-28T16:14:48.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.783+0000 7f4c4569c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c40071920 0x7f4c4013e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:48.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.783+0000 7f4c4569c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c40071920 0x7f4c4013e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.83:42604/0 (socket says 172.21.15.83:42604) 2024-03-28T16:14:48.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.783+0000 7f4c4569c640 1 -- 172.21.15.83:0/2488715349 learned_addr learned my addr 172.21.15.83:0/2488715349 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-28T16:14:48.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.783+0000 7f4c4569c640 10 monclient(hunting): get_auth_request con 0x7f4c40071920 auth_method 0 2024-03-28T16:14:48.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.783+0000 7f4c4569c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:48.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.783+0000 7f4c4569c640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:48.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.783+0000 7f4c4569c640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:48.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.783+0000 7f4c44e9b640 1 --2- 172.21.15.83:0/2488715349 >> v2:172.21.15.7:3300/0 conn(0x7f4c400a89a0 0x7f4c4013ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:48.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.783+0000 7f4c44e9b640 10 monclient(hunting): get_auth_request con 0x7f4c400a89a0 auth_method 0 2024-03-28T16:14:48.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.783+0000 7f4c44e9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:48.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.783+0000 7f4c44e9b640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:48.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.783+0000 7f4c44e9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:48.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.784+0000 7f4c4569c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:48.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.784+0000 7f4c4569c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:48.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.784+0000 7f4c4569c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:48.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.784+0000 7f4c44e9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:48.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.784+0000 7f4c44e9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:48.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.784+0000 7f4c44e9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:48.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.786+0000 7f4c4569c640 10 monclient(hunting): handle_auth_done global_id 4145 payload 274 2024-03-28T16:14:48.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.786+0000 7f4c4569c640 1 -- 172.21.15.83:0/2488715349 >> v2:172.21.15.7:3300/0 conn(0x7f4c400a89a0 msgr2=0x7f4c4013ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:14:48.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.786+0000 7f4c4569c640 1 --2- 172.21.15.83:0/2488715349 >> v2:172.21.15.7:3300/0 conn(0x7f4c400a89a0 0x7f4c4013ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:48.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.786+0000 7f4c4569c640 1 -- 172.21.15.83:0/2488715349 >> v1:172.21.15.7:6789/0 conn(0x7f4c400b1890 legacy=0x7f4c400b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:48.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.786+0000 7f4c4569c640 10 monclient: _finish_hunting 0 2024-03-28T16:14:48.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.786+0000 7f4c4569c640 1 monclient: found mon.noname-b 2024-03-28T16:14:48.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.786+0000 7f4c4569c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:48.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.786+0000 7f4c4569c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-03-28T16:14:48.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.786+0000 7f4c4569c640 1 -- 172.21.15.83:0/2488715349 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c400aa660 con 0x7f4c40071920 2024-03-28T16:14:48.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.788+0000 7f4c4569c640 1 --2- 172.21.15.83:0/2488715349 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c40071920 0x7f4c4013e830 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4c30009870 tx=0x7f4c3002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16668f63f505f789 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:48.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c37fff640 1 -- 172.21.15.83:0/2488715349 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c30030840 con 0x7f4c40071920 2024-03-28T16:14:48.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:48.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c37fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:14:48.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c37fff640 10 monclient: dump: 2024-03-28T16:14:48.793 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:48.793 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:48.793 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:48.793 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:48.793 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:48.793 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:48.793 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:48.794 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c37fff640 10 monclient: _finish_auth 0 2024-03-28T16:14:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c37fff640 10 monclient: _check_auth_tickets 2024-03-28T16:14:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c37fff640 1 -- 172.21.15.83:0/2488715349 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c300309a0 con 0x7f4c40071920 2024-03-28T16:14:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c47900640 5 monclient: authenticate success, global_id 4145 2024-03-28T16:14:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c47900640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:14:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c37fff640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c37fff640 1 -- 172.21.15.83:0/2488715349 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c30030d30 con 0x7f4c40071920 2024-03-28T16:14:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c47900640 10 monclient: get_monmap_and_config success 2024-03-28T16:14:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c47900640 4 set_mon_vals no callback set 2024-03-28T16:14:48.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c37fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:14:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c37fff640 10 monclient: dump: 2024-03-28T16:14:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:48.795 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:48.795 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:48.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c47900640 10 monclient: shutdown 2024-03-28T16:14:48.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c47900640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:48.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c47900640 1 -- 172.21.15.83:0/2488715349 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c40071920 msgr2=0x7f4c4013e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:48.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c47900640 1 --2- 172.21.15.83:0/2488715349 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c40071920 0x7f4c4013e830 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4c30009870 tx=0x7f4c3002b980 comp rx=0 tx=0).stop 2024-03-28T16:14:48.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c47900640 1 -- 172.21.15.83:0/2488715349 shutdown_connections 2024-03-28T16:14:48.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c47900640 1 --2- 172.21.15.83:0/2488715349 >> v2:172.21.15.7:3300/0 conn(0x7f4c400a89a0 0x7f4c4013ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:48.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.792+0000 7f4c47900640 1 --2- 172.21.15.83:0/2488715349 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c40071920 0x7f4c4013e830 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:48.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.793+0000 7f4c47900640 1 -- 172.21.15.83:0/2488715349 >> 172.21.15.83:0/2488715349 conn(0x7f4c400b3db0 msgr2=0x7f4c400b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:48.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.793+0000 7f4c47900640 1 -- 172.21.15.83:0/2488715349 shutdown_connections 2024-03-28T16:14:48.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.793+0000 7f4c47900640 1 -- 172.21.15.83:0/2488715349 wait complete. 2024-03-28T16:14:48.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.793+0000 7f4c47900640 10 monclient: build_initial_monmap 2024-03-28T16:14:48.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.793+0000 7f4c47900640 10 monclient: monmap: 2024-03-28T16:14:48.799 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-28T16:14:48.799 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:14:48.800 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-28T16:14:48.800 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-28T16:14:48.800 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:48.800 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:48.800 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:14:48.800 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:14:48.801 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:14:48.801 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:48.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.793+0000 7f4c47900640 1 librados: starting msgr at 2024-03-28T16:14:48.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.793+0000 7f4c47900640 1 librados: starting objecter 2024-03-28T16:14:48.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.794+0000 7f4c47900640 1 Processor -- start 2024-03-28T16:14:48.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.794+0000 7f4c47900640 1 -- start start 2024-03-28T16:14:48.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.794+0000 7f4c47900640 1 librados: setting wanted keys 2024-03-28T16:14:48.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.794+0000 7f4c47900640 1 librados: calling monclient init 2024-03-28T16:14:48.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.794+0000 7f4c47900640 10 monclient: init 2024-03-28T16:14:48.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.794+0000 7f4c47900640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:48.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.794+0000 7f4c47900640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-28T16:14:48.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.794+0000 7f4c47900640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c40071920 0x7f4c4013a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:48.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.794+0000 7f4c47900640 10 monclient(hunting): picked mon.noname-a con 0x7f4c40071920 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:14:48.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.794+0000 7f4c47900640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c400b1890 0x7f4c40149e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:48.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.794+0000 7f4c47900640 10 monclient(hunting): picked mon.noname-b con 0x7f4c400b1890 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:14:48.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.794+0000 7f4c47900640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c40147830 0x7f4c40148240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:48.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.794+0000 7f4c47900640 10 monclient(hunting): picked mon.noname-c con 0x7f4c40147830 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:48.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.794+0000 7f4c47900640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:48.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.794+0000 7f4c47900640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4014afb0 con 0x7f4c40071920 2024-03-28T16:14:48.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.794+0000 7f4c47900640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:48.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.794+0000 7f4c47900640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4014a380 con 0x7f4c400b1890 2024-03-28T16:14:48.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.794+0000 7f4c47900640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:48.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.794+0000 7f4c47900640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c40147db0 con 0x7f4c40147830 2024-03-28T16:14:48.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.794+0000 7f4c47900640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:48.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.794+0000 7f4c47900640 10 monclient(hunting): authenticate will time out at 1355.964431s 2024-03-28T16:14:48.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.795+0000 7f4c4569c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c40071920 0x7f4c4013a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:48.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.795+0000 7f4c44e9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c400b1890 0x7f4c40149e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:48.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.795+0000 7f4c45e9d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c40147830 0x7f4c40148240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:48.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.795+0000 7f4c4569c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c40071920 0x7f4c4013a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.83:51294/0 (socket says 172.21.15.83:51294) 2024-03-28T16:14:48.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.795+0000 7f4c4569c640 1 -- 172.21.15.83:0/1763537247 learned_addr learned my addr 172.21.15.83:0/1763537247 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-28T16:14:48.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.795+0000 7f4c4569c640 10 monclient(hunting): get_auth_request con 0x7f4c40071920 auth_method 0 2024-03-28T16:14:48.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.795+0000 7f4c4569c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:48.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.795+0000 7f4c4569c640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:48.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.795+0000 7f4c4569c640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:48.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.795+0000 7f4c45e9d640 10 monclient(hunting): get_auth_request con 0x7f4c40147830 auth_method 0 2024-03-28T16:14:48.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.795+0000 7f4c45e9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:48.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.795+0000 7f4c45e9d640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:48.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.795+0000 7f4c45e9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:48.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.795+0000 7f4c44e9b640 10 monclient(hunting): get_auth_request con 0x7f4c400b1890 auth_method 0 2024-03-28T16:14:48.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.795+0000 7f4c44e9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:48.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.795+0000 7f4c44e9b640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:48.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.795+0000 7f4c44e9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:48.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.795+0000 7f4c4569c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:48.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.795+0000 7f4c4569c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:48.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.795+0000 7f4c4569c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:48.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.796+0000 7f4c44e9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:48.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.796+0000 7f4c44e9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:48.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.796+0000 7f4c44e9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:48.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.796+0000 7f4c45e9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:48.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.796+0000 7f4c45e9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:48.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.796+0000 7f4c45e9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:48.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.798+0000 7f4c4569c640 10 monclient(hunting): handle_auth_done global_id 4146 payload 824 2024-03-28T16:14:48.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.798+0000 7f4c4569c640 1 -- 172.21.15.83:0/1763537247 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c40147830 msgr2=0x7f4c40148240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:48.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.798+0000 7f4c4569c640 1 --2- 172.21.15.83:0/1763537247 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c40147830 0x7f4c40148240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:48.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.798+0000 7f4c4569c640 1 -- 172.21.15.83:0/1763537247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c400b1890 msgr2=0x7f4c40149e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:48.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.798+0000 7f4c4569c640 1 --2- 172.21.15.83:0/1763537247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c400b1890 0x7f4c40149e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:48.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.798+0000 7f4c4569c640 10 monclient: _finish_hunting 0 2024-03-28T16:14:48.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.798+0000 7f4c4569c640 1 monclient: found mon.noname-a 2024-03-28T16:14:48.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.798+0000 7f4c4569c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:48.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.798+0000 7f4c4569c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:14:48.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.798+0000 7f4c4569c640 1 -- 172.21.15.83:0/1763537247 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c300094d0 con 0x7f4c40071920 2024-03-28T16:14:48.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.801+0000 7f4c4569c640 1 --2- 172.21.15.83:0/1763537247 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c40071920 0x7f4c4013a7b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4c3002b9c0 tx=0x7f4c30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:48.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.801+0000 7f4c35ffb640 10 client.?.objecter ms_handle_connect 0x7f4c40071920 2024-03-28T16:14:48.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.801+0000 7f4c35ffb640 10 client.?.objecter resend_mon_ops 2024-03-28T16:14:48.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.804+0000 7f4c35ffb640 1 -- 172.21.15.83:0/1763537247 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c30039c90 con 0x7f4c40071920 2024-03-28T16:14:48.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.804+0000 7f4c35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:48.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.804+0000 7f4c35ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:14:48.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.804+0000 7f4c35ffb640 10 monclient: dump: 2024-03-28T16:14:48.810 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:48.810 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:48.810 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:48.810 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:48.810 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:48.810 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:48.810 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:48.810 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:48.810 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:48.810 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:48.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.804+0000 7f4c35ffb640 10 monclient: _finish_auth 0 2024-03-28T16:14:48.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.804+0000 7f4c35ffb640 10 monclient: _check_auth_tickets 2024-03-28T16:14:48.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.804+0000 7f4c35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:48.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.804+0000 7f4c35ffb640 1 -- 172.21.15.83:0/1763537247 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c30039df0 con 0x7f4c40071920 2024-03-28T16:14:48.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.804+0000 7f4c35ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:48.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.804+0000 7f4c35ffb640 1 -- 172.21.15.83:0/1763537247 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c300446c0 con 0x7f4c40071920 2024-03-28T16:14:48.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.804+0000 7f4c35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:48.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.804+0000 7f4c35ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:14:48.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.804+0000 7f4c35ffb640 10 monclient: dump: 2024-03-28T16:14:48.811 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:48.811 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:48.811 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:48.811 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:48.811 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:48.811 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:48.812 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:48.812 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:48.812 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:48.812 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:48.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.804+0000 7f4c47900640 5 monclient: authenticate success, global_id 4146 2024-03-28T16:14:48.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.804+0000 7f4c47900640 10 monclient: _renew_subs 2024-03-28T16:14:48.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.804+0000 7f4c47900640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:14:48.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.804+0000 7f4c47900640 1 -- 172.21.15.83:0/1763537247 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c401497a0 con 0x7f4c40071920 2024-03-28T16:14:48.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.804+0000 7f4c377fe640 4 set_mon_vals no callback set 2024-03-28T16:14:48.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.805+0000 7f4c47900640 10 client.4146.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:14:48.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.805+0000 7f4c47900640 10 monclient: _renew_subs 2024-03-28T16:14:48.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.805+0000 7f4c47900640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:14:48.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.805+0000 7f4c47900640 1 -- 172.21.15.83:0/1763537247 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c40148780 con 0x7f4c40071920 2024-03-28T16:14:48.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.805+0000 7f4c47900640 1 librados: init done 2024-03-28T16:14:48.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.805+0000 7f4c47900640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:48.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.805+0000 7f4c47900640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:48.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.805+0000 7f4c47900640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:14:48.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.805+0000 7f4c47900640 1 -- 172.21.15.83:0/1763537247 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c40069430 con 0x7f4c40071920 2024-03-28T16:14:48.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.905+0000 7f4c35ffb640 1 -- 172.21.15.83:0/1763537247 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4c30044b20 con 0x7f4c40071920 2024-03-28T16:14:48.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.909+0000 7f4c35ffb640 1 -- 172.21.15.83:0/1763537247 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f4c3003e070 con 0x7f4c40071920 2024-03-28T16:14:48.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.909+0000 7f4c35ffb640 10 client.4146.objecter ms_dispatch 0x7f4c40000b90 osd_map(7..7 src has 1..7) v4 2024-03-28T16:14:48.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.909+0000 7f4c35ffb640 3 client.4146.objecter handle_osd_map got epochs [7,7] > 0 2024-03-28T16:14:48.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.909+0000 7f4c35ffb640 3 client.4146.objecter handle_osd_map decoding full epoch 7 2024-03-28T16:14:48.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.909+0000 7f4c35ffb640 20 client.4146.objecter dump_active .. 0 homeless 2024-03-28T16:14:48.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.909+0000 7f4c35ffb640 1 -- 172.21.15.83:0/1763537247 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f4c30049070 con 0x7f4c40071920 2024-03-28T16:14:48.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.909+0000 7f4c35ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:48.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.909+0000 7f4c35ffb640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:14:48.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.981+0000 7f4c47900640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2aabdbd2-d870-4d6d-823e-89b008ea3108", "id": 5}] 2024-03-28T16:14:48.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.981+0000 7f4c47900640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2aabdbd2-d870-4d6d-823e-89b008ea3108", "id": 5}] 2024-03-28T16:14:48.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.981+0000 7f4c47900640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:14:48.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:48.981+0000 7f4c47900640 1 -- 172.21.15.83:0/1763537247 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2aabdbd2-d870-4d6d-823e-89b008ea3108", "id": 5} v 0) v1 -- 0x7f4c40148970 con 0x7f4c40071920 2024-03-28T16:14:49.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.031+0000 7f4c35ffb640 1 -- 172.21.15.83:0/1763537247 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2aabdbd2-d870-4d6d-823e-89b008ea3108", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4c30034070 con 0x7f4c40071920 2024-03-28T16:14:49.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.031+0000 7f4c35ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2aabdbd2-d870-4d6d-823e-89b008ea3108", "id": 5}] 2024-03-28T16:14:49.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.031+0000 7f4c35ffb640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:14:49.032 INFO:teuthology.orchestra.run.smithi083.stdout:5 2024-03-28T16:14:49.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.031+0000 7f4c47900640 10 librados: watch_flush enter 2024-03-28T16:14:49.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.031+0000 7f4c47900640 10 librados: watch_flush exit 2024-03-28T16:14:49.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.031+0000 7f4c47900640 20 client.4146.objecter shutdown clearing up homeless session... 2024-03-28T16:14:49.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.031+0000 7f4c47900640 10 client.4146.objecter successfully canceled tick 2024-03-28T16:14:49.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.032+0000 7f4c47900640 10 monclient: shutdown 2024-03-28T16:14:49.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.032+0000 7f4c47900640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.032+0000 7f4c47900640 1 -- 172.21.15.83:0/1763537247 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c40071920 msgr2=0x7f4c4013a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.032+0000 7f4c47900640 1 --2- 172.21.15.83:0/1763537247 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c40071920 0x7f4c4013a7b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4c3002b9c0 tx=0x7f4c30004730 comp rx=0 tx=0).stop 2024-03-28T16:14:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.032+0000 7f4c47900640 1 -- 172.21.15.83:0/1763537247 shutdown_connections 2024-03-28T16:14:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.032+0000 7f4c47900640 1 --2- 172.21.15.83:0/1763537247 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4c40147830 0x7f4c40148240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.032+0000 7f4c47900640 1 --2- 172.21.15.83:0/1763537247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c400b1890 0x7f4c40149e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.032+0000 7f4c47900640 1 --2- 172.21.15.83:0/1763537247 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c40071920 0x7f4c4013a7b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.032+0000 7f4c47900640 1 -- 172.21.15.83:0/1763537247 >> 172.21.15.83:0/1763537247 conn(0x7f4c400b3db0 msgr2=0x7f4c4013e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.032+0000 7f4c47900640 1 -- 172.21.15.83:0/1763537247 shutdown_connections 2024-03-28T16:14:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.032+0000 7f4c47900640 1 -- 172.21.15.83:0/1763537247 wait complete. 2024-03-28T16:14:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.032+0000 7f4c47900640 1 librados: shutdown 2024-03-28T16:14:49.044 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd new fbbb7557-08d4-48d4-9b88-6bb1d680a87d 6 2024-03-28T16:14:49.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.142+0000 7f58f70fb640 10 monclient: get_monmap_and_config 2024-03-28T16:14:49.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.142+0000 7f58f70fb640 10 monclient: build_initial_monmap 2024-03-28T16:14:49.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.142+0000 7f58f70fb640 10 monclient: monmap: 2024-03-28T16:14:49.143 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-28T16:14:49.143 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:49.143 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:14:49.143642+0000 2024-03-28T16:14:49.143 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:14:49.143642+0000 2024-03-28T16:14:49.143 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:49.143 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:49.143 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:14:49.143 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:14:49.144 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:14:49.144 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:14:49.144 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:14:49.144 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:49.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.142+0000 7f58f70fb640 1 Processor -- start 2024-03-28T16:14:49.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.142+0000 7f58f70fb640 1 -- start start 2024-03-28T16:14:49.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.142+0000 7f58f70fb640 10 monclient: init 2024-03-28T16:14:49.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.143+0000 7f58f70fb640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:49.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.143+0000 7f58f70fb640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-03-28T16:14:49.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.143+0000 7f58f70fb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f58f0071480 addr v1:172.21.15.7:6789/0 2024-03-28T16:14:49.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.143+0000 7f58f70fb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f58f00b32e0 0x7f58f0144a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:49.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.143+0000 7f58f70fb640 10 monclient(hunting): picked mon.noname-a con 0x7f58f00b32e0 addr v2:172.21.15.7:3300/0 2024-03-28T16:14:49.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.143+0000 7f58f70fb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f58f00a86e0 0x7f58f0144f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:49.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.143+0000 7f58f70fb640 10 monclient(hunting): picked mon.noname-b con 0x7f58f00a86e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:49.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.143+0000 7f58f70fb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f00a8c90 con 0x7f58f0071480 2024-03-28T16:14:49.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.143+0000 7f58f70fb640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:49.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.143+0000 7f58f70fb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58f0071860 con 0x7f58f00b32e0 2024-03-28T16:14:49.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.143+0000 7f58f70fb640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:49.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.143+0000 7f58f70fb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f013f020 con 0x7f58f00a86e0 2024-03-28T16:14:49.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.143+0000 7f58f70fb640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:49.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.143+0000 7f58f70fb640 10 monclient(hunting): authenticate will time out at 1356.313355s 2024-03-28T16:14:49.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.144+0000 7f58f5698640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f58f00a86e0 0x7f58f0144f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:49.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.144+0000 7f58f5698640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f58f00a86e0 0x7f58f0144f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.83:42608/0 (socket says 172.21.15.83:42608) 2024-03-28T16:14:49.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.144+0000 7f58effff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f58f00b32e0 0x7f58f0144a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:49.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.144+0000 7f58f5698640 1 -- 172.21.15.83:0/3918087504 learned_addr learned my addr 172.21.15.83:0/3918087504 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-28T16:14:49.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.144+0000 7f58f5698640 10 monclient(hunting): get_auth_request con 0x7f58f00a86e0 auth_method 0 2024-03-28T16:14:49.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.144+0000 7f58f5698640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:49.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.144+0000 7f58f5698640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:49.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.144+0000 7f58f5698640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:49.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.144+0000 7f58effff640 10 monclient(hunting): get_auth_request con 0x7f58f00b32e0 auth_method 0 2024-03-28T16:14:49.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.144+0000 7f58effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:49.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.144+0000 7f58effff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:49.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.144+0000 7f58effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:49.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.144+0000 7f58f5698640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:49.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.144+0000 7f58f5698640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:49.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.144+0000 7f58f5698640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:49.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.145+0000 7f58effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:49.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.145+0000 7f58effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:49.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.145+0000 7f58effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:49.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.146+0000 7f58f5698640 10 monclient(hunting): handle_auth_done global_id 4151 payload 274 2024-03-28T16:14:49.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.146+0000 7f58f5698640 1 -- 172.21.15.83:0/3918087504 >> v2:172.21.15.7:3300/0 conn(0x7f58f00b32e0 msgr2=0x7f58f0144a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:14:49.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.146+0000 7f58f5698640 1 --2- 172.21.15.83:0/3918087504 >> v2:172.21.15.7:3300/0 conn(0x7f58f00b32e0 0x7f58f0144a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:49.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.146+0000 7f58f5698640 1 -- 172.21.15.83:0/3918087504 >> v1:172.21.15.7:6789/0 conn(0x7f58f0071480 legacy=0x7f58f013e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:49.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.146+0000 7f58f5698640 10 monclient: _finish_hunting 0 2024-03-28T16:14:49.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.146+0000 7f58f5698640 1 monclient: found mon.noname-b 2024-03-28T16:14:49.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.146+0000 7f58f5698640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:49.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.146+0000 7f58f5698640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-03-28T16:14:49.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.146+0000 7f58f5698640 1 -- 172.21.15.83:0/3918087504 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f0072140 con 0x7f58f00a86e0 2024-03-28T16:14:49.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.148+0000 7f58f5698640 1 --2- 172.21.15.83:0/3918087504 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f58f00a86e0 0x7f58f0144f40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f58e0009870 tx=0x7f58e002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56bcd6b03473c017 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:49.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.151+0000 7f58ef7fe640 1 -- 172.21.15.83:0/3918087504 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e00317c0 con 0x7f58f00a86e0 2024-03-28T16:14:49.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.151+0000 7f58ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:49.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.151+0000 7f58ef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:14:49.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.151+0000 7f58ef7fe640 10 monclient: dump: 2024-03-28T16:14:49.152 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:49.152 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:49.152 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:49.152 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:49.152 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:49.152 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:49.152 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:49.153 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:49.153 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:49.153 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:49.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.151+0000 7f58ef7fe640 10 monclient: _finish_auth 0 2024-03-28T16:14:49.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.151+0000 7f58ef7fe640 10 monclient: _check_auth_tickets 2024-03-28T16:14:49.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.151+0000 7f58ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:49.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.151+0000 7f58f70fb640 5 monclient: authenticate success, global_id 4151 2024-03-28T16:14:49.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.151+0000 7f58f70fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:14:49.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.153+0000 7f58ef7fe640 1 -- 172.21.15.83:0/3918087504 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e0031920 con 0x7f58f00a86e0 2024-03-28T16:14:49.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.153+0000 7f58ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:49.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.153+0000 7f58ef7fe640 1 -- 172.21.15.83:0/3918087504 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e0031cb0 con 0x7f58f00a86e0 2024-03-28T16:14:49.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.153+0000 7f58ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:49.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.153+0000 7f58ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:14:49.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.153+0000 7f58ef7fe640 10 monclient: dump: 2024-03-28T16:14:49.156 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:49.156 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:49.156 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:49.156 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:49.156 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:49.156 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:49.156 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:49.156 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:49.156 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:49.157 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:49.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.153+0000 7f58f70fb640 10 monclient: get_monmap_and_config success 2024-03-28T16:14:49.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.153+0000 7f58f70fb640 4 set_mon_vals no callback set 2024-03-28T16:14:49.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.154+0000 7f58f70fb640 10 monclient: shutdown 2024-03-28T16:14:49.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.154+0000 7f58f70fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:49.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.154+0000 7f58f70fb640 1 -- 172.21.15.83:0/3918087504 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f58f00a86e0 msgr2=0x7f58f0144f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:49.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.154+0000 7f58f70fb640 1 --2- 172.21.15.83:0/3918087504 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f58f00a86e0 0x7f58f0144f40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f58e0009870 tx=0x7f58e002b860 comp rx=0 tx=0).stop 2024-03-28T16:14:49.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.154+0000 7f58f70fb640 1 -- 172.21.15.83:0/3918087504 shutdown_connections 2024-03-28T16:14:49.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.154+0000 7f58f70fb640 1 --2- 172.21.15.83:0/3918087504 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f58f00a86e0 0x7f58f0144f40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:49.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.154+0000 7f58f70fb640 1 --2- 172.21.15.83:0/3918087504 >> v2:172.21.15.7:3300/0 conn(0x7f58f00b32e0 0x7f58f0144a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:49.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.154+0000 7f58f70fb640 1 -- 172.21.15.83:0/3918087504 >> 172.21.15.83:0/3918087504 conn(0x7f58f00b70e0 msgr2=0x7f58f00b74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:49.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.154+0000 7f58f70fb640 1 -- 172.21.15.83:0/3918087504 shutdown_connections 2024-03-28T16:14:49.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.154+0000 7f58f70fb640 1 -- 172.21.15.83:0/3918087504 wait complete. 2024-03-28T16:14:49.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.154+0000 7f58f70fb640 10 monclient: build_initial_monmap 2024-03-28T16:14:49.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.154+0000 7f58f70fb640 10 monclient: monmap: 2024-03-28T16:14:49.158 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-28T16:14:49.158 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:14:49.158 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-28T16:14:49.158 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-28T16:14:49.158 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:49.158 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:49.158 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:14:49.158 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:14:49.158 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:14:49.158 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:49.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 1 librados: starting msgr at 2024-03-28T16:14:49.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 1 librados: starting objecter 2024-03-28T16:14:49.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 1 Processor -- start 2024-03-28T16:14:49.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 1 -- start start 2024-03-28T16:14:49.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 1 librados: setting wanted keys 2024-03-28T16:14:49.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 1 librados: calling monclient init 2024-03-28T16:14:49.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 10 monclient: init 2024-03-28T16:14:49.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:49.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-28T16:14:49.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58f0071480 0x7f58f013ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:49.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 10 monclient(hunting): picked mon.noname-b con 0x7f58f0071480 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:14:49.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f58f00a86e0 0x7f58f0143b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:49.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 10 monclient(hunting): picked mon.noname-c con 0x7f58f00a86e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:49.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58f00b32e0 0x7f58f0144060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:49.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 10 monclient(hunting): picked mon.noname-a con 0x7f58f00b32e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:14:49.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:49.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f01348c0 con 0x7f58f00b32e0 2024-03-28T16:14:49.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:49.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f013d8b0 con 0x7f58f0071480 2024-03-28T16:14:49.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:49.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f013c410 con 0x7f58f00a86e0 2024-03-28T16:14:49.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:49.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.155+0000 7f58f70fb640 10 monclient(hunting): authenticate will time out at 1356.325834s 2024-03-28T16:14:49.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.156+0000 7f58effff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f58f00a86e0 0x7f58f0143b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:49.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.156+0000 7f58f5698640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58f00b32e0 0x7f58f0144060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:49.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.156+0000 7f58f4e97640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58f0071480 0x7f58f013ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:49.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.156+0000 7f58f5698640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58f00b32e0 0x7f58f0144060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.83:51312/0 (socket says 172.21.15.83:51312) 2024-03-28T16:14:49.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.156+0000 7f58effff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f58f00a86e0 0x7f58f0143b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.83:42616/0 (socket says 172.21.15.83:42616) 2024-03-28T16:14:49.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.156+0000 7f58f5698640 1 -- 172.21.15.83:0/2726829705 learned_addr learned my addr 172.21.15.83:0/2726829705 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-28T16:14:49.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.156+0000 7f58f5698640 10 monclient(hunting): get_auth_request con 0x7f58f00b32e0 auth_method 0 2024-03-28T16:14:49.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.156+0000 7f58f5698640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:49.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.156+0000 7f58f5698640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:49.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.156+0000 7f58f5698640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:49.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.156+0000 7f58effff640 10 monclient(hunting): get_auth_request con 0x7f58f00a86e0 auth_method 0 2024-03-28T16:14:49.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.156+0000 7f58effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:49.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.156+0000 7f58effff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:49.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.156+0000 7f58effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:49.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.156+0000 7f58f4e97640 10 monclient(hunting): get_auth_request con 0x7f58f0071480 auth_method 0 2024-03-28T16:14:49.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.156+0000 7f58f4e97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:49.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.156+0000 7f58f4e97640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:49.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.156+0000 7f58f4e97640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:49.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.157+0000 7f58f5698640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:49.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.157+0000 7f58f5698640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:49.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.157+0000 7f58f5698640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:49.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.157+0000 7f58effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:49.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.157+0000 7f58effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:49.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.157+0000 7f58effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:49.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.157+0000 7f58f4e97640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:49.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.157+0000 7f58f4e97640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:49.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.157+0000 7f58f4e97640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:49.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.160+0000 7f58f4e97640 10 monclient(hunting): handle_auth_done global_id 4150 payload 824 2024-03-28T16:14:49.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.160+0000 7f58f4e97640 1 -- 172.21.15.83:0/2726829705 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f58f00a86e0 msgr2=0x7f58f0143b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:49.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.160+0000 7f58f4e97640 1 --2- 172.21.15.83:0/2726829705 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f58f00a86e0 0x7f58f0143b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:49.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.160+0000 7f58f4e97640 1 -- 172.21.15.83:0/2726829705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58f00b32e0 msgr2=0x7f58f0144060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:49.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.160+0000 7f58f4e97640 1 --2- 172.21.15.83:0/2726829705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58f00b32e0 0x7f58f0144060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:49.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.160+0000 7f58f4e97640 10 monclient: _finish_hunting 0 2024-03-28T16:14:49.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.160+0000 7f58f4e97640 1 monclient: found mon.noname-b 2024-03-28T16:14:49.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.160+0000 7f58f4e97640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:49.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.160+0000 7f58f4e97640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:14:49.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.160+0000 7f58f4e97640 1 -- 172.21.15.83:0/2726829705 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e00094d0 con 0x7f58f0071480 2024-03-28T16:14:49.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.160+0000 7f58f5698640 1 --2- 172.21.15.83:0/2726829705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58f00b32e0 0x7f58f0144060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:14:49.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.162+0000 7f58f4e97640 1 --2- 172.21.15.83:0/2726829705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58f0071480 0x7f58f013ddd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f58e800c770 tx=0x7f58e800cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:49.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.162+0000 7f58ed7fa640 10 client.?.objecter ms_handle_connect 0x7f58f0071480 2024-03-28T16:14:49.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.162+0000 7f58ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-28T16:14:49.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58ed7fa640 1 -- 172.21.15.83:0/2726829705 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e8003b70 con 0x7f58f0071480 2024-03-28T16:14:49.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:49.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58ed7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:14:49.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58ed7fa640 10 monclient: dump: 2024-03-28T16:14:49.167 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:49.167 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:49.167 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:49.167 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:49.168 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:49.168 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:49.168 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:49.168 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:49.168 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:49.168 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:49.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58ed7fa640 10 monclient: _finish_auth 0 2024-03-28T16:14:49.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58ed7fa640 10 monclient: _check_auth_tickets 2024-03-28T16:14:49.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:49.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58ed7fa640 1 -- 172.21.15.83:0/2726829705 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e8003cd0 con 0x7f58f0071480 2024-03-28T16:14:49.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58f70fb640 5 monclient: authenticate success, global_id 4150 2024-03-28T16:14:49.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58ed7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:49.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58f70fb640 10 monclient: _renew_subs 2024-03-28T16:14:49.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58ed7fa640 1 -- 172.21.15.83:0/2726829705 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e800aa30 con 0x7f58f0071480 2024-03-28T16:14:49.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58f70fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:49.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58f70fb640 1 -- 172.21.15.83:0/2726829705 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f013d4a0 con 0x7f58f0071480 2024-03-28T16:14:49.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58eeffd640 4 set_mon_vals no callback set 2024-03-28T16:14:49.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:49.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58ed7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:14:49.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58ed7fa640 10 monclient: dump: 2024-03-28T16:14:49.169 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:49.169 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:49.169 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:49.169 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:49.169 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:49.169 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:49.169 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:49.169 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:49.169 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:49.169 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:49.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58f70fb640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:14:49.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58f70fb640 10 monclient: _renew_subs 2024-03-28T16:14:49.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58f70fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:49.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58f70fb640 1 -- 172.21.15.83:0/2726829705 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f0144780 con 0x7f58f0071480 2024-03-28T16:14:49.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.165+0000 7f58f70fb640 1 librados: init done 2024-03-28T16:14:49.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.166+0000 7f58f70fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:49.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.166+0000 7f58f70fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:49.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.166+0000 7f58f70fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:49.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.166+0000 7f58f70fb640 1 -- 172.21.15.83:0/2726829705 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58f013da70 con 0x7f58f0071480 2024-03-28T16:14:49.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.168+0000 7f58ed7fa640 1 -- 172.21.15.83:0/2726829705 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f58e80050d0 con 0x7f58f0071480 2024-03-28T16:14:49.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.168+0000 7f58ed7fa640 1 -- 172.21.15.83:0/2726829705 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f58e8025a20 con 0x7f58f0071480 2024-03-28T16:14:49.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.168+0000 7f58ed7fa640 10 client.4150.objecter ms_dispatch 0x7f58f0000b90 osd_map(8..8 src has 1..8) v4 2024-03-28T16:14:49.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.168+0000 7f58ed7fa640 3 client.4150.objecter handle_osd_map got epochs [8,8] > 0 2024-03-28T16:14:49.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.168+0000 7f58ed7fa640 3 client.4150.objecter handle_osd_map decoding full epoch 8 2024-03-28T16:14:49.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.168+0000 7f58ed7fa640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-28T16:14:49.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.282+0000 7f58ed7fa640 1 -- 172.21.15.83:0/2726829705 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f58e8025c00 con 0x7f58f0071480 2024-03-28T16:14:49.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.282+0000 7f58ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:49.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.282+0000 7f58ed7fa640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:14:49.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.354+0000 7f58f70fb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fbbb7557-08d4-48d4-9b88-6bb1d680a87d", "id": 6}] 2024-03-28T16:14:49.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.354+0000 7f58f70fb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fbbb7557-08d4-48d4-9b88-6bb1d680a87d", "id": 6}] 2024-03-28T16:14:49.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.354+0000 7f58f70fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:49.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.354+0000 7f58f70fb640 1 -- 172.21.15.83:0/2726829705 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fbbb7557-08d4-48d4-9b88-6bb1d680a87d", "id": 6} v 0) v1 -- 0x7f58f0067040 con 0x7f58f0071480 2024-03-28T16:14:49.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.437+0000 7f58ed7fa640 1 -- 172.21.15.83:0/2726829705 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fbbb7557-08d4-48d4-9b88-6bb1d680a87d", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f58e8025da0 con 0x7f58f0071480 2024-03-28T16:14:49.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.437+0000 7f58ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fbbb7557-08d4-48d4-9b88-6bb1d680a87d", "id": 6}] 2024-03-28T16:14:49.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.437+0000 7f58ed7fa640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:14:49.439 INFO:teuthology.orchestra.run.smithi083.stdout:6 2024-03-28T16:14:49.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.438+0000 7f58f70fb640 10 librados: watch_flush enter 2024-03-28T16:14:49.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.438+0000 7f58f70fb640 10 librados: watch_flush exit 2024-03-28T16:14:49.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.438+0000 7f58f70fb640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-28T16:14:49.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.438+0000 7f58f70fb640 10 client.4150.objecter successfully canceled tick 2024-03-28T16:14:49.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.438+0000 7f58f70fb640 10 monclient: shutdown 2024-03-28T16:14:49.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.438+0000 7f58f70fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:49.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.438+0000 7f58f70fb640 1 -- 172.21.15.83:0/2726829705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58f0071480 msgr2=0x7f58f013ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:49.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.438+0000 7f58f70fb640 1 --2- 172.21.15.83:0/2726829705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58f0071480 0x7f58f013ddd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f58e800c770 tx=0x7f58e800cc30 comp rx=0 tx=0).stop 2024-03-28T16:14:49.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.438+0000 7f58f70fb640 1 -- 172.21.15.83:0/2726829705 shutdown_connections 2024-03-28T16:14:49.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.438+0000 7f58f70fb640 1 --2- 172.21.15.83:0/2726829705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58f00b32e0 0x7f58f0144060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:49.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.438+0000 7f58f70fb640 1 --2- 172.21.15.83:0/2726829705 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f58f00a86e0 0x7f58f0143b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:49.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.438+0000 7f58f70fb640 1 --2- 172.21.15.83:0/2726829705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58f0071480 0x7f58f013ddd0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:49.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.438+0000 7f58f70fb640 1 -- 172.21.15.83:0/2726829705 >> 172.21.15.83:0/2726829705 conn(0x7f58f00b70e0 msgr2=0x7f58f00b4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:49.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.438+0000 7f58f70fb640 1 -- 172.21.15.83:0/2726829705 shutdown_connections 2024-03-28T16:14:49.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.438+0000 7f58f70fb640 1 -- 172.21.15.83:0/2726829705 wait complete. 2024-03-28T16:14:49.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.438+0000 7f58f70fb640 1 librados: shutdown 2024-03-28T16:14:49.449 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd new 805f54d9-03d5-4b23-aef3-e1d55bb09f87 7 2024-03-28T16:14:49.483 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:49.482+0000 7f32d7241040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T16:14:49.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.549+0000 7ff20b98e640 10 monclient: get_monmap_and_config 2024-03-28T16:14:49.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.549+0000 7ff20b98e640 10 monclient: build_initial_monmap 2024-03-28T16:14:49.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.549+0000 7ff20b98e640 10 monclient: monmap: 2024-03-28T16:14:49.550 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-28T16:14:49.551 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:49.551 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:14:49.550561+0000 2024-03-28T16:14:49.551 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:14:49.550561+0000 2024-03-28T16:14:49.551 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:49.551 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:49.551 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:14:49.551 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:14:49.551 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:14:49.551 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:14:49.551 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:14:49.552 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:49.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.549+0000 7ff20b98e640 1 Processor -- start 2024-03-28T16:14:49.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.549+0000 7ff20b98e640 1 -- start start 2024-03-28T16:14:49.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.549+0000 7ff20b98e640 10 monclient: init 2024-03-28T16:14:49.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.550+0000 7ff20b98e640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:49.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.550+0000 7ff20b98e640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-03-28T16:14:49.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.550+0000 7ff20b98e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff204071480 0x7ff20413e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:49.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.550+0000 7ff20b98e640 10 monclient(hunting): picked mon.noname-b con 0x7ff204071480 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:49.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.550+0000 7ff20b98e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff2040b32e0 0x7ff204144a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:49.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.550+0000 7ff20b98e640 10 monclient(hunting): picked mon.noname-c con 0x7ff2040b32e0 addr v2:172.21.15.83:3300/0 2024-03-28T16:14:49.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.550+0000 7ff20b98e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff20413ee50 addr v1:172.21.15.7:6789/0 2024-03-28T16:14:49.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.550+0000 7ff20b98e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2040a8c90 con 0x7ff20413ee50 2024-03-28T16:14:49.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.550+0000 7ff20b98e640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:49.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.550+0000 7ff20b98e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff204071860 con 0x7ff2040b32e0 2024-03-28T16:14:49.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.550+0000 7ff20b98e640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:49.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.550+0000 7ff20b98e640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff204145650 con 0x7ff204071480 2024-03-28T16:14:49.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.550+0000 7ff20b98e640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.550+0000 7ff20b98e640 10 monclient(hunting): authenticate will time out at 1356.720324s 2024-03-28T16:14:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.550+0000 7ff208f29640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff2040b32e0 0x7ff204144a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.551+0000 7ff209f2b640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff20413ee50 0x7ff204144f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.83:45732/0 (socket says 172.21.15.83:45732) 2024-03-28T16:14:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.551+0000 7ff209f2b640 1 -- 172.21.15.83:0/1899873927 learned_addr learned my addr 172.21.15.83:0/1899873927 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-28T16:14:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.551+0000 7ff20972a640 1 --2- 172.21.15.83:0/1899873927 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff204071480 0x7ff20413e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.551+0000 7ff208f29640 10 monclient(hunting): get_auth_request con 0x7ff2040b32e0 auth_method 0 2024-03-28T16:14:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.551+0000 7ff208f29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.551+0000 7ff208f29640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.551+0000 7ff208f29640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.551+0000 7ff20972a640 10 monclient(hunting): get_auth_request con 0x7ff204071480 auth_method 0 2024-03-28T16:14:49.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.551+0000 7ff20972a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:49.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.551+0000 7ff20972a640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:49.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.551+0000 7ff20972a640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:49.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.551+0000 7ff20972a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:49.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.551+0000 7ff20972a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:49.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.551+0000 7ff20972a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:49.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.551+0000 7ff208f29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:49.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.551+0000 7ff208f29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:49.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.551+0000 7ff208f29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:49.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.553+0000 7ff208f29640 10 monclient(hunting): handle_auth_done global_id 4153 payload 274 2024-03-28T16:14:49.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.553+0000 7ff208f29640 1 -- 172.21.15.83:0/1899873927 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff204071480 msgr2=0x7ff20413e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:14:49.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.553+0000 7ff208f29640 1 --2- 172.21.15.83:0/1899873927 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff204071480 0x7ff20413e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:49.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.553+0000 7ff208f29640 1 -- 172.21.15.83:0/1899873927 >> v1:172.21.15.7:6789/0 conn(0x7ff20413ee50 legacy=0x7ff204144f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:49.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.553+0000 7ff208f29640 10 monclient: _finish_hunting 0 2024-03-28T16:14:49.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.553+0000 7ff208f29640 1 monclient: found mon.noname-c 2024-03-28T16:14:49.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.553+0000 7ff208f29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:49.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.553+0000 7ff208f29640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3300/0 2024-03-28T16:14:49.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.553+0000 7ff208f29640 1 -- 172.21.15.83:0/1899873927 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff204072140 con 0x7ff2040b32e0 2024-03-28T16:14:49.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.557+0000 7ff208f29640 1 --2- 172.21.15.83:0/1899873927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2040b32e0 0x7ff204144a00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff1f40097a0 tx=0x7ff1f402b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3695dd1c77a59ed server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:49.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.561+0000 7ff1fbfff640 1 -- 172.21.15.83:0/1899873927 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f402f050 con 0x7ff2040b32e0 2024-03-28T16:14:49.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.561+0000 7ff1fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:49.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.561+0000 7ff1fbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:14:49.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.561+0000 7ff1fbfff640 10 monclient: dump: 2024-03-28T16:14:49.564 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:49.564 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:49.564 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:49.564 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:49.565 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:49.565 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:49.565 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:49.565 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:49.565 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:49.565 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:49.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.561+0000 7ff1fbfff640 10 monclient: _finish_auth 0 2024-03-28T16:14:49.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.561+0000 7ff1fbfff640 10 monclient: _check_auth_tickets 2024-03-28T16:14:49.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.561+0000 7ff1fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:49.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.561+0000 7ff1fbfff640 1 -- 172.21.15.83:0/1899873927 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f4031700 con 0x7ff2040b32e0 2024-03-28T16:14:49.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.561+0000 7ff1fbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:49.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.561+0000 7ff1fbfff640 1 -- 172.21.15.83:0/1899873927 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f4031a90 con 0x7ff2040b32e0 2024-03-28T16:14:49.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.561+0000 7ff20b98e640 5 monclient: authenticate success, global_id 4153 2024-03-28T16:14:49.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.561+0000 7ff20b98e640 10 monclient: get_monmap_and_config success 2024-03-28T16:14:49.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.561+0000 7ff20b98e640 4 set_mon_vals no callback set 2024-03-28T16:14:49.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.561+0000 7ff1fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:49.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.561+0000 7ff1fbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:14:49.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.561+0000 7ff1fbfff640 10 monclient: dump: 2024-03-28T16:14:49.566 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:49.566 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:49.567 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:49.567 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:49.567 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:49.567 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:49.567 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:49.567 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:49.567 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:49.567 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:49.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.562+0000 7ff20b98e640 10 monclient: shutdown 2024-03-28T16:14:49.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.562+0000 7ff20b98e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:49.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.562+0000 7ff20b98e640 1 -- 172.21.15.83:0/1899873927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2040b32e0 msgr2=0x7ff204144a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:49.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.562+0000 7ff20b98e640 1 --2- 172.21.15.83:0/1899873927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2040b32e0 0x7ff204144a00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff1f40097a0 tx=0x7ff1f402b7a0 comp rx=0 tx=0).stop 2024-03-28T16:14:49.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.562+0000 7ff20b98e640 1 -- 172.21.15.83:0/1899873927 shutdown_connections 2024-03-28T16:14:49.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.562+0000 7ff20b98e640 1 --2- 172.21.15.83:0/1899873927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2040b32e0 0x7ff204144a00 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:49.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.562+0000 7ff20b98e640 1 --2- 172.21.15.83:0/1899873927 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff204071480 0x7ff20413e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:49.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.562+0000 7ff20b98e640 1 -- 172.21.15.83:0/1899873927 >> 172.21.15.83:0/1899873927 conn(0x7ff2040b70e0 msgr2=0x7ff2040b74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:49.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.562+0000 7ff20b98e640 1 -- 172.21.15.83:0/1899873927 shutdown_connections 2024-03-28T16:14:49.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.562+0000 7ff20b98e640 1 -- 172.21.15.83:0/1899873927 wait complete. 2024-03-28T16:14:49.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.563+0000 7ff20b98e640 10 monclient: build_initial_monmap 2024-03-28T16:14:49.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.563+0000 7ff20b98e640 10 monclient: monmap: 2024-03-28T16:14:49.568 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-28T16:14:49.568 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:14:49.568 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-28T16:14:49.568 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-28T16:14:49.568 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:49.568 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:49.568 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:14:49.568 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:14:49.568 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:14:49.568 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:49.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.563+0000 7ff20b98e640 1 librados: starting msgr at 2024-03-28T16:14:49.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.563+0000 7ff20b98e640 1 librados: starting objecter 2024-03-28T16:14:49.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.563+0000 7ff20b98e640 1 Processor -- start 2024-03-28T16:14:49.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.563+0000 7ff20b98e640 1 -- start start 2024-03-28T16:14:49.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.563+0000 7ff20b98e640 1 librados: setting wanted keys 2024-03-28T16:14:49.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.563+0000 7ff20b98e640 1 librados: calling monclient init 2024-03-28T16:14:49.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.563+0000 7ff20b98e640 10 monclient: init 2024-03-28T16:14:49.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.563+0000 7ff20b98e640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:49.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.563+0000 7ff20b98e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-28T16:14:49.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.563+0000 7ff20b98e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff204071480 0x7ff2041d6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:49.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff20b98e640 10 monclient(hunting): picked mon.noname-c con 0x7ff204071480 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:49.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff20b98e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2040b32e0 0x7ff2041d7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:49.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff20b98e640 10 monclient(hunting): picked mon.noname-a con 0x7ff2040b32e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:14:49.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff20b98e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff20413ee50 0x7ff204144240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:49.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff20b98e640 10 monclient(hunting): picked mon.noname-b con 0x7ff20413ee50 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:14:49.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff20b98e640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:49.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff20b98e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff204144780 con 0x7ff2040b32e0 2024-03-28T16:14:49.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff20b98e640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:49.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff20b98e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2041d6b60 con 0x7ff20413ee50 2024-03-28T16:14:49.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff20b98e640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:49.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff20b98e640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff204143900 con 0x7ff204071480 2024-03-28T16:14:49.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff20b98e640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:49.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff20b98e640 10 monclient(hunting): authenticate will time out at 1356.733927s 2024-03-28T16:14:49.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff209f2b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff20413ee50 0x7ff204144240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:49.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff208f29640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2040b32e0 0x7ff2041d7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:49.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff20972a640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff204071480 0x7ff2041d6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:49.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff208f29640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2040b32e0 0x7ff2041d7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.83:51326/0 (socket says 172.21.15.83:51326) 2024-03-28T16:14:49.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff208f29640 1 -- 172.21.15.83:0/2640475883 learned_addr learned my addr 172.21.15.83:0/2640475883 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-28T16:14:49.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff208f29640 10 monclient(hunting): get_auth_request con 0x7ff2040b32e0 auth_method 0 2024-03-28T16:14:49.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff208f29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:49.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff208f29640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:49.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff208f29640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:49.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff20972a640 10 monclient(hunting): get_auth_request con 0x7ff204071480 auth_method 0 2024-03-28T16:14:49.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff20972a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:49.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff20972a640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:49.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff20972a640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:49.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff209f2b640 10 monclient(hunting): get_auth_request con 0x7ff20413ee50 auth_method 0 2024-03-28T16:14:49.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff209f2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:49.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff209f2b640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:49.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.564+0000 7ff209f2b640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:49.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.565+0000 7ff208f29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:49.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.565+0000 7ff208f29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:49.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.565+0000 7ff208f29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:49.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.565+0000 7ff20972a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:49.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.565+0000 7ff20972a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:49.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.565+0000 7ff20972a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:49.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.565+0000 7ff209f2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:49.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.565+0000 7ff209f2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:49.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.565+0000 7ff209f2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:49.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.568+0000 7ff208f29640 10 monclient(hunting): handle_auth_done global_id 4158 payload 824 2024-03-28T16:14:49.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.568+0000 7ff208f29640 1 -- 172.21.15.83:0/2640475883 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff204071480 msgr2=0x7ff2041d6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:49.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.568+0000 7ff208f29640 1 --2- 172.21.15.83:0/2640475883 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff204071480 0x7ff2041d6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:49.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.568+0000 7ff208f29640 1 -- 172.21.15.83:0/2640475883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff20413ee50 msgr2=0x7ff204144240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:49.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.568+0000 7ff208f29640 1 --2- 172.21.15.83:0/2640475883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff20413ee50 0x7ff204144240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:49.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.568+0000 7ff208f29640 10 monclient: _finish_hunting 0 2024-03-28T16:14:49.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.568+0000 7ff208f29640 1 monclient: found mon.noname-a 2024-03-28T16:14:49.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.568+0000 7ff208f29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:49.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.568+0000 7ff208f29640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:14:49.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.568+0000 7ff208f29640 1 -- 172.21.15.83:0/2640475883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f4009400 con 0x7ff2040b32e0 2024-03-28T16:14:49.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.570+0000 7ff208f29640 1 --2- 172.21.15.83:0/2640475883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2040b32e0 0x7ff2041d7080 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff1f402bdb0 tx=0x7ff1f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:49.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.570+0000 7ff1f9ffb640 10 client.?.objecter ms_handle_connect 0x7ff2040b32e0 2024-03-28T16:14:49.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.570+0000 7ff1f9ffb640 10 client.?.objecter resend_mon_ops 2024-03-28T16:14:49.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.573+0000 7ff1f9ffb640 1 -- 172.21.15.83:0/2640475883 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f4033bb0 con 0x7ff2040b32e0 2024-03-28T16:14:49.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.573+0000 7ff1f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:49.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.574+0000 7ff1f9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:14:49.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.574+0000 7ff1f9ffb640 10 monclient: dump: 2024-03-28T16:14:49.581 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:49.581 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:49.581 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:49.581 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:49.581 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:49.581 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:49.581 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:49.582 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:49.582 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:49.582 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:49.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.574+0000 7ff1f9ffb640 10 monclient: _finish_auth 0 2024-03-28T16:14:49.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.574+0000 7ff1f9ffb640 10 monclient: _check_auth_tickets 2024-03-28T16:14:49.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.574+0000 7ff1f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:49.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.574+0000 7ff1f9ffb640 1 -- 172.21.15.83:0/2640475883 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f4033d30 con 0x7ff2040b32e0 2024-03-28T16:14:49.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.574+0000 7ff1f9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:49.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.574+0000 7ff1f9ffb640 1 -- 172.21.15.83:0/2640475883 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f4033ea0 con 0x7ff2040b32e0 2024-03-28T16:14:49.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.574+0000 7ff20b98e640 5 monclient: authenticate success, global_id 4158 2024-03-28T16:14:49.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.574+0000 7ff20b98e640 10 monclient: _renew_subs 2024-03-28T16:14:49.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.574+0000 7ff20b98e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:14:49.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.574+0000 7ff20b98e640 1 -- 172.21.15.83:0/2640475883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff204143b20 con 0x7ff2040b32e0 2024-03-28T16:14:49.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.574+0000 7ff1fb7fe640 4 set_mon_vals no callback set 2024-03-28T16:14:49.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.574+0000 7ff1f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:49.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.574+0000 7ff1f9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:14:49.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.574+0000 7ff1f9ffb640 10 monclient: dump: 2024-03-28T16:14:49.584 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-28T16:14:49.584 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:49.584 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:49.584 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:49.584 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:49.584 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-28T16:14:49.584 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:49.584 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:49.584 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:49.585 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-28T16:14:49.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.574+0000 7ff20b98e640 10 client.4158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:14:49.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.574+0000 7ff20b98e640 10 monclient: _renew_subs 2024-03-28T16:14:49.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.574+0000 7ff20b98e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:14:49.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.574+0000 7ff20b98e640 1 -- 172.21.15.83:0/2640475883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff204143f80 con 0x7ff2040b32e0 2024-03-28T16:14:49.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.574+0000 7ff20b98e640 1 librados: init done 2024-03-28T16:14:49.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.575+0000 7ff20b98e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:49.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.575+0000 7ff20b98e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.575+0000 7ff20b98e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:14:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.575+0000 7ff20b98e640 1 -- 172.21.15.83:0/2640475883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2040b6c70 con 0x7ff2040b32e0 2024-03-28T16:14:49.586 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:49.578+0000 7f392f430040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T16:14:49.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.675+0000 7ff1f9ffb640 1 -- 172.21.15.83:0/2640475883 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff1f403a050 con 0x7ff2040b32e0 2024-03-28T16:14:49.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.679+0000 7ff1f9ffb640 1 -- 172.21.15.83:0/2640475883 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7ff1f4051a90 con 0x7ff2040b32e0 2024-03-28T16:14:49.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.679+0000 7ff1f9ffb640 10 client.4158.objecter ms_dispatch 0x7ff204000b90 osd_map(9..9 src has 1..9) v4 2024-03-28T16:14:49.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.680+0000 7ff1f9ffb640 3 client.4158.objecter handle_osd_map got epochs [9,9] > 0 2024-03-28T16:14:49.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.680+0000 7ff1f9ffb640 3 client.4158.objecter handle_osd_map decoding full epoch 9 2024-03-28T16:14:49.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.680+0000 7ff1f9ffb640 20 client.4158.objecter dump_active .. 0 homeless 2024-03-28T16:14:49.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.680+0000 7ff1f9ffb640 1 -- 172.21.15.83:0/2640475883 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7ff1f4043070 con 0x7ff2040b32e0 2024-03-28T16:14:49.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.680+0000 7ff1f9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:49.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.680+0000 7ff1f9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:14:49.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.750+0000 7ff20b98e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "805f54d9-03d5-4b23-aef3-e1d55bb09f87", "id": 7}] 2024-03-28T16:14:49.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.751+0000 7ff20b98e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "805f54d9-03d5-4b23-aef3-e1d55bb09f87", "id": 7}] 2024-03-28T16:14:49.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.751+0000 7ff20b98e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:14:49.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.751+0000 7ff20b98e640 1 -- 172.21.15.83:0/2640475883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd new", "uuid": "805f54d9-03d5-4b23-aef3-e1d55bb09f87", "id": 7} v 0) v1 -- 0x7ff2041d6d20 con 0x7ff2040b32e0 2024-03-28T16:14:49.764 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:49.763+0000 7f32d7241040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T16:14:49.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.801+0000 7ff1f9ffb640 1 -- 172.21.15.83:0/2640475883 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "805f54d9-03d5-4b23-aef3-e1d55bb09f87", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff1f4038070 con 0x7ff2040b32e0 2024-03-28T16:14:49.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.801+0000 7ff1f9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "805f54d9-03d5-4b23-aef3-e1d55bb09f87", "id": 7}] 2024-03-28T16:14:49.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.801+0000 7ff1f9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:14:49.802 INFO:teuthology.orchestra.run.smithi083.stdout:7 2024-03-28T16:14:49.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.801+0000 7ff20b98e640 10 librados: watch_flush enter 2024-03-28T16:14:49.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.801+0000 7ff20b98e640 10 librados: watch_flush exit 2024-03-28T16:14:49.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.801+0000 7ff20b98e640 20 client.4158.objecter shutdown clearing up homeless session... 2024-03-28T16:14:49.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.801+0000 7ff20b98e640 10 client.4158.objecter successfully canceled tick 2024-03-28T16:14:49.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.802+0000 7ff20b98e640 10 monclient: shutdown 2024-03-28T16:14:49.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.802+0000 7ff20b98e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:49.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.802+0000 7ff20b98e640 1 -- 172.21.15.83:0/2640475883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2040b32e0 msgr2=0x7ff2041d7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:49.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.802+0000 7ff20b98e640 1 --2- 172.21.15.83:0/2640475883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2040b32e0 0x7ff2041d7080 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff1f402bdb0 tx=0x7ff1f40046c0 comp rx=0 tx=0).stop 2024-03-28T16:14:49.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.802+0000 7ff20b98e640 1 -- 172.21.15.83:0/2640475883 shutdown_connections 2024-03-28T16:14:49.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.802+0000 7ff20b98e640 1 --2- 172.21.15.83:0/2640475883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2040b32e0 0x7ff2041d7080 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:49.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.802+0000 7ff20b98e640 1 --2- 172.21.15.83:0/2640475883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff20413ee50 0x7ff204144240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:49.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.802+0000 7ff20b98e640 1 --2- 172.21.15.83:0/2640475883 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff204071480 0x7ff2041d6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:49.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.802+0000 7ff20b98e640 1 -- 172.21.15.83:0/2640475883 >> 172.21.15.83:0/2640475883 conn(0x7ff2040b70e0 msgr2=0x7ff2040b4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:49.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.802+0000 7ff20b98e640 1 -- 172.21.15.83:0/2640475883 shutdown_connections 2024-03-28T16:14:49.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.802+0000 7ff20b98e640 1 -- 172.21.15.83:0/2640475883 wait complete. 2024-03-28T16:14:49.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-28T16:14:49.802+0000 7ff20b98e640 1 librados: shutdown 2024-03-28T16:14:49.813 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-28T16:14:49.813 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-28T16:14:49.813 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-28T16:14:49.815 INFO:tasks.ceph.osd.0:Started 2024-03-28T16:14:49.816 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-28T16:14:49.816 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-28T16:14:49.816 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-28T16:14:49.817 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:49.816+0000 7f32d7241040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T16:14:49.818 INFO:tasks.ceph.osd.1:Started 2024-03-28T16:14:49.818 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-28T16:14:49.818 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-28T16:14:49.819 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-28T16:14:49.821 INFO:tasks.ceph.osd.2:Started 2024-03-28T16:14:49.821 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-28T16:14:49.821 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-28T16:14:49.821 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-28T16:14:49.823 INFO:tasks.ceph.osd.3:Started 2024-03-28T16:14:49.823 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-28T16:14:49.823 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-28T16:14:49.823 DEBUG:teuthology.orchestra.run.smithi083:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-28T16:14:49.826 INFO:tasks.ceph.osd.4:Started 2024-03-28T16:14:49.826 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-28T16:14:49.826 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-28T16:14:49.829 DEBUG:teuthology.orchestra.run.smithi083:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-28T16:14:49.831 INFO:tasks.ceph.osd.5:Started 2024-03-28T16:14:49.831 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-28T16:14:49.832 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-28T16:14:49.832 DEBUG:teuthology.orchestra.run.smithi083:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-28T16:14:49.834 INFO:tasks.ceph.osd.6:Started 2024-03-28T16:14:49.834 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-28T16:14:49.834 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-28T16:14:49.834 DEBUG:teuthology.orchestra.run.smithi083:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-28T16:14:49.836 INFO:tasks.ceph.osd.7:Started 2024-03-28T16:14:49.837 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:14:49.885 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:49.884+0000 7f392f430040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T16:14:49.973 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:49.971+0000 7f392f430040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T16:14:49.981 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:49.980+0000 7f32d7241040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T16:14:49.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.994+0000 7f3520496640 10 monclient: get_monmap_and_config 2024-03-28T16:14:49.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.994+0000 7f3520496640 10 monclient: build_initial_monmap 2024-03-28T16:14:49.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.995+0000 7f3520496640 10 monclient: monmap: 2024-03-28T16:14:49.996 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:14:49.996 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:49.997 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:14:49.996628+0000 2024-03-28T16:14:49.997 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:14:49.996628+0000 2024-03-28T16:14:49.997 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:49.997 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:49.997 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:14:49.997 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:14:49.997 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:14:49.997 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:14:49.997 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:14:49.997 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:14:50.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.995+0000 7f3520496640 1 Processor -- start 2024-03-28T16:14:50.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.996+0000 7f3520496640 1 -- start start 2024-03-28T16:14:50.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.996+0000 7f3520496640 10 monclient: init 2024-03-28T16:14:50.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.997+0000 7f3520496640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:50.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.997+0000 7f3520496640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-03-28T16:14:50.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.997+0000 7f3520496640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3518071480 0x7f351813e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:50.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.997+0000 7f3520496640 10 monclient(hunting): picked mon.noname-a con 0x7f3518071480 addr v2:172.21.15.7:3300/0 2024-03-28T16:14:50.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.997+0000 7f3520496640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35180b32e0 addr v1:172.21.15.7:6789/0 2024-03-28T16:14:50.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.997+0000 7f3520496640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f351813ee50 0x7f3518145110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:50.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.997+0000 7f3520496640 10 monclient(hunting): picked mon.noname-c con 0x7f351813ee50 addr v2:172.21.15.83:3300/0 2024-03-28T16:14:50.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.997+0000 7f3520496640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35180a8c90 con 0x7f35180b32e0 2024-03-28T16:14:50.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.997+0000 7f3520496640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:50.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.997+0000 7f3520496640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3518071860 con 0x7f3518071480 2024-03-28T16:14:50.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.997+0000 7f3520496640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:50.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.997+0000 7f3520496640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3518145650 con 0x7f351813ee50 2024-03-28T16:14:50.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.997+0000 7f3520496640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:50.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.997+0000 7f3520496640 10 monclient(hunting): authenticate will time out at 1357.139684s 2024-03-28T16:14:50.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.998+0000 7f351da31640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f35180b32e0 0x7f3518144a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53848/0 (socket says 172.21.15.7:53848) 2024-03-28T16:14:50.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.998+0000 7f351da31640 1 -- 172.21.15.7:0/1292138490 learned_addr learned my addr 172.21.15.7:0/1292138490 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-28T16:14:50.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.998+0000 7f351e232640 1 --2- 172.21.15.7:0/1292138490 >> v2:172.21.15.7:3300/0 conn(0x7f3518071480 0x7f351813e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:50.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.999+0000 7f351e232640 10 monclient(hunting): get_auth_request con 0x7f3518071480 auth_method 0 2024-03-28T16:14:50.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.999+0000 7f351e232640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:50.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.999+0000 7f351e232640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:50.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.999+0000 7f351e232640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:50.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.999+0000 7f351e232640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:50.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.999+0000 7f351e232640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:50.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:49.999+0000 7f351e232640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:50.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.002+0000 7f351e232640 10 monclient(hunting): handle_auth_done global_id 4161 payload 274 2024-03-28T16:14:50.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.002+0000 7f351e232640 1 -- 172.21.15.7:0/1292138490 >> v2:172.21.15.83:3300/0 conn(0x7f351813ee50 msgr2=0x7f3518145110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:14:50.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.002+0000 7f351e232640 1 --2- 172.21.15.7:0/1292138490 >> v2:172.21.15.83:3300/0 conn(0x7f351813ee50 0x7f3518145110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:50.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.002+0000 7f351e232640 1 -- 172.21.15.7:0/1292138490 >> v1:172.21.15.7:6789/0 conn(0x7f35180b32e0 legacy=0x7f3518144a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:50.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.002+0000 7f351e232640 10 monclient: _finish_hunting 0 2024-03-28T16:14:50.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.002+0000 7f351e232640 1 monclient: found mon.noname-a 2024-03-28T16:14:50.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.002+0000 7f351e232640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:50.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.002+0000 7f351e232640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:14:50.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.002+0000 7f351e232640 1 -- 172.21.15.7:0/1292138490 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3518072140 con 0x7f3518071480 2024-03-28T16:14:50.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.007+0000 7f351e232640 1 --2- 172.21.15.7:0/1292138490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3518071480 0x7f351813e910 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f35180a94b0 tx=0x7f351402b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=722b3db6a115f79b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:50.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.013+0000 7f351d230640 1 -- 172.21.15.7:0/1292138490 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3514031520 con 0x7f3518071480 2024-03-28T16:14:50.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.013+0000 7f351d230640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:50.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.013+0000 7f351d230640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:14:50.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.013+0000 7f351d230640 10 monclient: dump: 2024-03-28T16:14:50.017 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:14:50.017 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:50.017 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:50.017 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:50.017 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:50.017 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:50.017 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:50.017 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:50.018 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:50.018 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:14:50.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.013+0000 7f351d230640 10 monclient: _finish_auth 0 2024-03-28T16:14:50.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.013+0000 7f351d230640 10 monclient: _check_auth_tickets 2024-03-28T16:14:50.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.013+0000 7f351d230640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:50.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.013+0000 7f351d230640 1 -- 172.21.15.7:0/1292138490 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3514031680 con 0x7f3518071480 2024-03-28T16:14:50.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.013+0000 7f351d230640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:50.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.013+0000 7f351d230640 1 -- 172.21.15.7:0/1292138490 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3514031a10 con 0x7f3518071480 2024-03-28T16:14:50.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.013+0000 7f3520496640 5 monclient: authenticate success, global_id 4161 2024-03-28T16:14:50.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.013+0000 7f3520496640 10 monclient: get_monmap_and_config success 2024-03-28T16:14:50.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.013+0000 7f3520496640 4 set_mon_vals no callback set 2024-03-28T16:14:50.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.013+0000 7f351d230640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:50.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.013+0000 7f351d230640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:14:50.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.013+0000 7f351d230640 10 monclient: dump: 2024-03-28T16:14:50.019 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:14:50.019 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:50.019 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:50.019 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:50.019 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:50.019 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:50.019 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:50.019 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:50.019 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:50.019 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:14:50.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.014+0000 7f3520496640 10 monclient: shutdown 2024-03-28T16:14:50.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.014+0000 7f3520496640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:50.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.014+0000 7f3520496640 1 -- 172.21.15.7:0/1292138490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3518071480 msgr2=0x7f351813e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:50.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.014+0000 7f3520496640 1 --2- 172.21.15.7:0/1292138490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3518071480 0x7f351813e910 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f35180a94b0 tx=0x7f351402b7c0 comp rx=0 tx=0).stop 2024-03-28T16:14:50.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.014+0000 7f3520496640 1 -- 172.21.15.7:0/1292138490 shutdown_connections 2024-03-28T16:14:50.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.014+0000 7f3520496640 1 --2- 172.21.15.7:0/1292138490 >> v2:172.21.15.83:3300/0 conn(0x7f351813ee50 0x7f3518145110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:50.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.014+0000 7f3520496640 1 --2- 172.21.15.7:0/1292138490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3518071480 0x7f351813e910 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:50.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.014+0000 7f3520496640 1 -- 172.21.15.7:0/1292138490 >> 172.21.15.7:0/1292138490 conn(0x7f35180b70e0 msgr2=0x7f35180b74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:50.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.014+0000 7f3520496640 1 -- 172.21.15.7:0/1292138490 shutdown_connections 2024-03-28T16:14:50.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.014+0000 7f3520496640 1 -- 172.21.15.7:0/1292138490 wait complete. 2024-03-28T16:14:50.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.016+0000 7f3520496640 10 monclient: build_initial_monmap 2024-03-28T16:14:50.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.016+0000 7f3520496640 10 monclient: monmap: 2024-03-28T16:14:50.020 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:14:50.020 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:14:50.020 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:14:50.020 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:14:50.020 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:50.020 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:50.020 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:14:50.020 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:14:50.020 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:14:50.021 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:14:50.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.016+0000 7f3520496640 1 librados: starting msgr at 2024-03-28T16:14:50.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.016+0000 7f3520496640 1 librados: starting objecter 2024-03-28T16:14:50.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.017+0000 7f3520496640 1 Processor -- start 2024-03-28T16:14:50.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.017+0000 7f3520496640 1 -- start start 2024-03-28T16:14:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.017+0000 7f3520496640 1 librados: setting wanted keys 2024-03-28T16:14:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.017+0000 7f3520496640 1 librados: calling monclient init 2024-03-28T16:14:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.017+0000 7f3520496640 10 monclient: init 2024-03-28T16:14:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.018+0000 7f3520496640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.018+0000 7f3520496640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-28T16:14:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.018+0000 7f3520496640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3518071480 0x7f3518143350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.018+0000 7f3520496640 10 monclient(hunting): picked mon.noname-b con 0x7f3518071480 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:14:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.018+0000 7f3520496640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35180b32e0 0x7f35181d6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.018+0000 7f3520496640 10 monclient(hunting): picked mon.noname-a con 0x7f35180b32e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:14:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.019+0000 7f3520496640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f351813ee50 0x7f35181d6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.019+0000 7f3520496640 10 monclient(hunting): picked mon.noname-c con 0x7f351813ee50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.019+0000 7f3520496640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.019+0000 7f3520496640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3518144600 con 0x7f35180b32e0 2024-03-28T16:14:50.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.019+0000 7f3520496640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:50.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.019+0000 7f3520496640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35181d7240 con 0x7f3518071480 2024-03-28T16:14:50.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.019+0000 7f351da31640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35180b32e0 0x7f35181d6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:50.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.019+0000 7f3520496640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:50.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.019+0000 7f3520496640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3518143920 con 0x7f351813ee50 2024-03-28T16:14:50.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.019+0000 7f3520496640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:50.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.019+0000 7f3520496640 10 monclient(hunting): authenticate will time out at 1357.161495s 2024-03-28T16:14:50.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.019+0000 7f351da31640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35180b32e0 0x7f35181d6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42026/0 (socket says 172.21.15.7:42026) 2024-03-28T16:14:50.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.019+0000 7f351da31640 1 -- 172.21.15.7:0/502907457 learned_addr learned my addr 172.21.15.7:0/502907457 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:14:50.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.019+0000 7f351da31640 10 monclient(hunting): get_auth_request con 0x7f35180b32e0 auth_method 0 2024-03-28T16:14:50.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.019+0000 7f351da31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:50.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.019+0000 7f351da31640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:50.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.019+0000 7f351da31640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:50.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.020+0000 7f351ea33640 1 --2- 172.21.15.7:0/502907457 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f351813ee50 0x7f35181d6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:50.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.020+0000 7f351da31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:50.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.020+0000 7f351da31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:50.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.020+0000 7f351da31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:50.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.020+0000 7f351ea33640 10 monclient(hunting): get_auth_request con 0x7f351813ee50 auth_method 0 2024-03-28T16:14:50.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.020+0000 7f351ea33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:50.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.020+0000 7f351ea33640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:50.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.020+0000 7f351ea33640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:50.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.021+0000 7f351ea33640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:50.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.021+0000 7f351ea33640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:50.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.021+0000 7f351ea33640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:50.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.024+0000 7f351da31640 10 monclient(hunting): handle_auth_done global_id 4164 payload 824 2024-03-28T16:14:50.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.024+0000 7f351da31640 1 -- 172.21.15.7:0/502907457 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f351813ee50 msgr2=0x7f35181d6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:50.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.024+0000 7f351da31640 1 --2- 172.21.15.7:0/502907457 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f351813ee50 0x7f35181d6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:50.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.024+0000 7f351da31640 1 -- 172.21.15.7:0/502907457 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3518071480 msgr2=0x7f3518143350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:50.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.024+0000 7f351da31640 1 --2- 172.21.15.7:0/502907457 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3518071480 0x7f3518143350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:50.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.024+0000 7f351da31640 10 monclient: _finish_hunting 0 2024-03-28T16:14:50.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.024+0000 7f351da31640 1 monclient: found mon.noname-a 2024-03-28T16:14:50.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.024+0000 7f351da31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:50.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.024+0000 7f351da31640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:14:50.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.024+0000 7f351da31640 1 -- 172.21.15.7:0/502907457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3514009400 con 0x7f35180b32e0 2024-03-28T16:14:50.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.029+0000 7f351da31640 1 --2- 172.21.15.7:0/502907457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35180b32e0 0x7f35181d6790 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f350800c4a0 tx=0x7f350800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:50.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.029+0000 7f350effd640 10 client.?.objecter ms_handle_connect 0x7f35180b32e0 2024-03-28T16:14:50.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.029+0000 7f350effd640 10 client.?.objecter resend_mon_ops 2024-03-28T16:14:50.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.034+0000 7f350effd640 1 -- 172.21.15.7:0/502907457 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3508005ce0 con 0x7f35180b32e0 2024-03-28T16:14:50.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.034+0000 7f350effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:50.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.034+0000 7f350effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:14:50.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.034+0000 7f350effd640 10 monclient: dump: 2024-03-28T16:14:50.037 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:14:50.037 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:50.038 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:50.038 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:50.038 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:50.038 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:50.038 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:50.038 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:50.038 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:50.038 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:14:50.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.034+0000 7f350effd640 10 monclient: _finish_auth 0 2024-03-28T16:14:50.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.034+0000 7f350effd640 10 monclient: _check_auth_tickets 2024-03-28T16:14:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.034+0000 7f350effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.034+0000 7f350effd640 1 -- 172.21.15.7:0/502907457 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3508005e40 con 0x7f35180b32e0 2024-03-28T16:14:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.034+0000 7f350effd640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.034+0000 7f351ca2f640 4 set_mon_vals no callback set 2024-03-28T16:14:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.035+0000 7f3520496640 5 monclient: authenticate success, global_id 4164 2024-03-28T16:14:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.035+0000 7f350effd640 1 -- 172.21.15.7:0/502907457 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3508007270 con 0x7f35180b32e0 2024-03-28T16:14:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.035+0000 7f350effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.035+0000 7f350effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:14:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.035+0000 7f350effd640 10 monclient: dump: 2024-03-28T16:14:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:14:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:50.039 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:50.040 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:14:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.035+0000 7f3520496640 10 monclient: _renew_subs 2024-03-28T16:14:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.035+0000 7f3520496640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:14:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.035+0000 7f3520496640 1 -- 172.21.15.7:0/502907457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f351813e320 con 0x7f35180b32e0 2024-03-28T16:14:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.035+0000 7f3520496640 10 client.4164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:14:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.035+0000 7f3520496640 10 monclient: _renew_subs 2024-03-28T16:14:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.035+0000 7f3520496640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:14:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.035+0000 7f3520496640 1 -- 172.21.15.7:0/502907457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35181d3fe0 con 0x7f35180b32e0 2024-03-28T16:14:50.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.035+0000 7f3520496640 1 librados: init done 2024-03-28T16:14:50.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.037+0000 7f3520496640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:50.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.037+0000 7f3520496640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:50.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.037+0000 7f3520496640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:14:50.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.037+0000 7f3520496640 1 -- 172.21.15.7:0/502907457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35181d64b0 con 0x7f35180b32e0 2024-03-28T16:14:50.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.039+0000 7f350effd640 1 -- 172.21.15.7:0/502907457 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f350801c440 con 0x7f35180b32e0 2024-03-28T16:14:50.184 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-28T16:14:50.183+0000 7f392f430040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T16:14:50.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.219+0000 7f350effd640 1 -- 172.21.15.7:0/502907457 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f350801b3b0 con 0x7f35180b32e0 2024-03-28T16:14:50.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.219+0000 7f350effd640 10 client.4164.objecter ms_dispatch 0x7f3518000b90 osd_map(10..10 src has 1..10) v4 2024-03-28T16:14:50.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.219+0000 7f350effd640 3 client.4164.objecter handle_osd_map got epochs [10,10] > 0 2024-03-28T16:14:50.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.219+0000 7f350effd640 3 client.4164.objecter handle_osd_map decoding full epoch 10 2024-03-28T16:14:50.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.219+0000 7f350effd640 20 client.4164.objecter dump_active .. 0 homeless 2024-03-28T16:14:50.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.219+0000 7f350effd640 1 -- 172.21.15.7:0/502907457 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f35181d64b0 con 0x7f35180b32e0 2024-03-28T16:14:50.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.219+0000 7f350effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:50.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.219+0000 7f350effd640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:14:50.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.342+0000 7f3520496640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:14:50.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.342+0000 7f3520496640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:14:50.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.342+0000 7f3520496640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:14:50.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.342+0000 7f3520496640 1 -- 172.21.15.7:0/502907457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3518067040 con 0x7f35180b32e0 2024-03-28T16:14:50.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.501+0000 7f350effd640 1 -- 172.21.15.7:0/502907457 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f3508021060 con 0x7f35180b32e0 2024-03-28T16:14:50.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.501+0000 7f350effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:14:50.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.501+0000 7f350effd640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:14:50.503 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:14:50.503 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":10,"fsid":"808ff1a1-2b17-4c35-a89a-340dbbf674ee","created":"2024-03-28T16:14:44.591599+0000","modified":"2024-03-28T16:14:49.757700+0000","last_up_change":"0.000000","last_in_change":"2024-03-28T16:14:49.757700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a19c6856-78b9-4919-97ce-6940a5af982c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2afee73d-3369-4fd2-bfc5-36dee610daf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0473d84c-e864-4373-b9d1-283130278363","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b64fd5e7-cdec-4bd0-b5e2-dfc8d4f6b9ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8f7552db-5651-430f-a826-3c6843bffb99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2aabdbd2-d870-4d6d-823e-89b008ea3108","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fbbb7557-08d4-48d4-9b88-6bb1d680a87d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"805f54d9-03d5-4b23-aef3-e1d55bb09f87","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:14:50.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.502+0000 7f3520496640 10 librados: watch_flush enter 2024-03-28T16:14:50.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.503+0000 7f3520496640 10 librados: watch_flush exit 2024-03-28T16:14:50.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.503+0000 7f3520496640 20 client.4164.objecter shutdown clearing up homeless session... 2024-03-28T16:14:50.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.503+0000 7f3520496640 10 client.4164.objecter successfully canceled tick 2024-03-28T16:14:50.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.503+0000 7f3520496640 10 monclient: shutdown 2024-03-28T16:14:50.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.503+0000 7f3520496640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:50.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.503+0000 7f3520496640 1 -- 172.21.15.7:0/502907457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35180b32e0 msgr2=0x7f35181d6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:50.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.503+0000 7f3520496640 1 --2- 172.21.15.7:0/502907457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35180b32e0 0x7f35181d6790 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f350800c4a0 tx=0x7f350800c960 comp rx=0 tx=0).stop 2024-03-28T16:14:50.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.503+0000 7f3520496640 1 -- 172.21.15.7:0/502907457 shutdown_connections 2024-03-28T16:14:50.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.503+0000 7f3520496640 1 --2- 172.21.15.7:0/502907457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35180b32e0 0x7f35181d6790 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:50.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.503+0000 7f3520496640 1 --2- 172.21.15.7:0/502907457 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f351813ee50 0x7f35181d6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:50.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.503+0000 7f3520496640 1 --2- 172.21.15.7:0/502907457 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3518071480 0x7f3518143350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:50.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.503+0000 7f3520496640 1 -- 172.21.15.7:0/502907457 >> 172.21.15.7:0/502907457 conn(0x7f35180b70e0 msgr2=0x7f35180b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:50.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.503+0000 7f3520496640 1 -- 172.21.15.7:0/502907457 shutdown_connections 2024-03-28T16:14:50.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.503+0000 7f3520496640 1 -- 172.21.15.7:0/502907457 wait complete. 2024-03-28T16:14:50.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:14:50.504+0000 7f3520496640 1 librados: shutdown 2024-03-28T16:14:50.520 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-28T16:14:50.521 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-28T16:14:50.967 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:50.966+0000 7f32d1a87640 -1 mgr handle_mgr_map I was active but no longer am 2024-03-28T16:14:51.327 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-28T16:14:51.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.470+0000 7fb10d3dd640 10 monclient: get_monmap_and_config 2024-03-28T16:14:51.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.470+0000 7fb10d3dd640 10 monclient: build_initial_monmap 2024-03-28T16:14:51.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.470+0000 7fb10d3dd640 10 monclient: monmap: 2024-03-28T16:14:51.472 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-28T16:14:51.472 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:51.472 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:14:51.472501+0000 2024-03-28T16:14:51.472 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:14:51.472501+0000 2024-03-28T16:14:51.472 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:51.472 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:51.472 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:14:51.472 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:14:51.472 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:14:51.473 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:14:51.473 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:14:51.473 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:14:51.475 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.471+0000 7fb10d3dd640 1 Processor -- start 2024-03-28T16:14:51.475 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.471+0000 7fb10d3dd640 1 -- start start 2024-03-28T16:14:51.475 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.471+0000 7fb10d3dd640 10 monclient: init 2024-03-28T16:14:51.475 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.472+0000 7fb10d3dd640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:51.475 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.472+0000 7fb10d3dd640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-03-28T16:14:51.475 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.472+0000 7fb10d3dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb1080b3ff0 addr v1:172.21.15.83:6789/0 2024-03-28T16:14:51.475 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.473+0000 7fb10d3dd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb1080b3680 0x7fb108145060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:51.475 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.473+0000 7fb10d3dd640 10 monclient(hunting): picked mon.noname-c con 0x7fb1080b3680 addr v2:172.21.15.83:3300/0 2024-03-28T16:14:51.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.473+0000 7fb10d3dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb1080a86e0 addr v1:172.21.15.7:6789/0 2024-03-28T16:14:51.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.473+0000 7fb10d3dd640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1080a9170 con 0x7fb1080a86e0 2024-03-28T16:14:51.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.473+0000 7fb10d3dd640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1080aa4b0 con 0x7fb1080b3ff0 2024-03-28T16:14:51.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.473+0000 7fb10d3dd640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:51.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.473+0000 7fb10d3dd640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1080b43d0 con 0x7fb1080b3680 2024-03-28T16:14:51.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.473+0000 7fb10d3dd640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:51.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.473+0000 7fb10d3dd640 10 monclient(hunting): authenticate will time out at 1358.615415s 2024-03-28T16:14:51.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.473+0000 7fb1077fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb1080a86e0 0x7fb10813e550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53856/0 (socket says 172.21.15.7:53856) 2024-03-28T16:14:51.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.473+0000 7fb1077fe640 1 -- 172.21.15.7:0/1100780309 learned_addr learned my addr 172.21.15.7:0/1100780309 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-28T16:14:51.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.474+0000 7fb1067fc640 1 --2- 172.21.15.7:0/1100780309 >> v2:172.21.15.83:3300/0 conn(0x7fb1080b3680 0x7fb108145060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:51.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.476+0000 7fb105ffb640 1 -- 172.21.15.7:0/1100780309 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232639437 0 0) 0x7fb1080a9170 con 0x7fb1080a86e0 2024-03-28T16:14:51.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.476+0000 7fb105ffb640 10 monclient(hunting): _negotiate 2024-03-28T16:14:51.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.476+0000 7fb105ffb640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:51.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.476+0000 7fb105ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:51.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.476+0000 7fb105ffb640 10 monclient(hunting): my global_id is 4173 2024-03-28T16:14:51.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.476+0000 7fb105ffb640 1 -- 172.21.15.7:0/1100780309 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0f0003520 con 0x7fb1080a86e0 2024-03-28T16:14:51.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.477+0000 7fb1067fc640 10 monclient(hunting): get_auth_request con 0x7fb1080b3680 auth_method 0 2024-03-28T16:14:51.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.477+0000 7fb1067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:51.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.477+0000 7fb1067fc640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:51.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.477+0000 7fb1067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:51.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.478+0000 7fb105ffb640 1 -- 172.21.15.7:0/1100780309 <== mon.1 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435993010 0 0) 0x7fb1080aa4b0 con 0x7fb1080b3ff0 2024-03-28T16:14:51.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.478+0000 7fb105ffb640 10 monclient(hunting): _negotiate 2024-03-28T16:14:51.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.478+0000 7fb105ffb640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:51.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.478+0000 7fb105ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:51.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.478+0000 7fb105ffb640 10 monclient(hunting): my global_id is 4165 2024-03-28T16:14:51.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.478+0000 7fb105ffb640 1 -- 172.21.15.7:0/1100780309 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1080a9170 con 0x7fb1080b3ff0 2024-03-28T16:14:51.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.479+0000 7fb1067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:51.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.479+0000 7fb1067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:51.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.479+0000 7fb1067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:51.490 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 1 -- 172.21.15.7:0/1100780309 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3751041356 0 0) 0x7fb0f0003520 con 0x7fb1080a86e0 2024-03-28T16:14:51.491 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 1 -- 172.21.15.7:0/1100780309 >> v2:172.21.15.83:3300/0 conn(0x7fb1080b3680 msgr2=0x7fb108145060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:14:51.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 1 --2- 172.21.15.7:0/1100780309 >> v2:172.21.15.83:3300/0 conn(0x7fb1080b3680 0x7fb108145060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:51.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 1 -- 172.21.15.7:0/1100780309 >> v1:172.21.15.83:6789/0 conn(0x7fb1080b3ff0 legacy=0x7fb108144950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:51.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 10 monclient: _finish_hunting 0 2024-03-28T16:14:51.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 1 monclient: found mon.noname-a-legacy 2024-03-28T16:14:51.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:51.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.7:6789/0 2024-03-28T16:14:51.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 1 -- 172.21.15.7:0/1100780309 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1080a8c50 con 0x7fb1080a86e0 2024-03-28T16:14:51.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 10 monclient: _finish_auth 0 2024-03-28T16:14:51.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 10 monclient: _check_auth_tickets 2024-03-28T16:14:51.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:51.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 1 -- 172.21.15.7:0/1100780309 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1105724391 0 0) 0x7fb1000019f0 con 0x7fb1080a86e0 2024-03-28T16:14:51.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:51.493 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-03-28T16:14:51.493 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 10 monclient: dump: 2024-03-28T16:14:51.493 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:14:51.493 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:51.493 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:51.493 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:51.493 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:51.493 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:51.493 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:51.493 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:51.493 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:51.493 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:14:51.493 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] but i'm connected to v1:172.21.15.7:6789/0, reconnecting 2024-03-28T16:14:51.493 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:51.493 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 1 -- 172.21.15.7:0/1100780309 >> v1:172.21.15.7:6789/0 conn(0x7fb1080a86e0 legacy=0x7fb10813e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:51.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-28T16:14:51.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 1 --2- 172.21.15.7:0/1100780309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0f0003b70 0x7fb0f0003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:51.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 10 monclient(hunting): picked mon.a con 0x7fb0f0003b70 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:14:51.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 1 --2- 172.21.15.7:0/1100780309 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0f00047c0 0x7fb0f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:51.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 10 monclient(hunting): picked mon.c con 0x7fb0f00047c0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:51.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 1 --2- 172.21.15.7:0/1100780309 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb0f0005180 0x7fb0f000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:51.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 10 monclient(hunting): picked mon.b con 0x7fb0f0005180 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:14:51.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:51.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 1 -- 172.21.15.7:0/1100780309 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1080b43d0 con 0x7fb0f0003b70 2024-03-28T16:14:51.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:51.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 1 -- 172.21.15.7:0/1100780309 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f0014890 con 0x7fb0f0005180 2024-03-28T16:14:51.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:51.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 1 -- 172.21.15.7:0/1100780309 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f0014a50 con 0x7fb0f00047c0 2024-03-28T16:14:51.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:51.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.482+0000 7fb105ffb640 10 monclient(hunting): _finish_auth 0 2024-03-28T16:14:51.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.483+0000 7fb106ffd640 1 --2- 172.21.15.7:0/1100780309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0f0003b70 0x7fb0f0003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:51.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.483+0000 7fb1067fc640 1 --2- 172.21.15.7:0/1100780309 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0f00047c0 0x7fb0f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:51.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.484+0000 7fb1067fc640 10 monclient(hunting): get_auth_request con 0x7fb0f00047c0 auth_method 0 2024-03-28T16:14:51.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.484+0000 7fb1067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:51.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.484+0000 7fb1067fc640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:51.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.484+0000 7fb1067fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:14:51.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.485+0000 7fb1067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:51.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.485+0000 7fb1067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:51.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.485+0000 7fb1067fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-28T16:14:51.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.485+0000 7fb106ffd640 10 monclient(hunting): get_auth_request con 0x7fb0f0003b70 auth_method 0 2024-03-28T16:14:51.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.485+0000 7fb106ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:51.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.485+0000 7fb106ffd640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:51.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.485+0000 7fb106ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:14:51.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.487+0000 7fb106ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:51.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.487+0000 7fb106ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:51.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.487+0000 7fb106ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-28T16:14:51.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.488+0000 7fb1067fc640 10 monclient(hunting): handle_auth_done global_id 4173 payload 293 2024-03-28T16:14:51.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.488+0000 7fb1067fc640 1 -- 172.21.15.7:0/1100780309 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb0f0005180 msgr2=0x7fb0f000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:14:51.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.488+0000 7fb1067fc640 1 --2- 172.21.15.7:0/1100780309 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb0f0005180 0x7fb0f000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:51.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.488+0000 7fb1067fc640 1 -- 172.21.15.7:0/1100780309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0f0003b70 msgr2=0x7fb0f0003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:14:51.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.488+0000 7fb1067fc640 1 --2- 172.21.15.7:0/1100780309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb0f0003b70 0x7fb0f0003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:51.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.488+0000 7fb1067fc640 10 monclient: _finish_hunting 0 2024-03-28T16:14:51.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.488+0000 7fb1067fc640 1 monclient: found mon.c 2024-03-28T16:14:51.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.488+0000 7fb1067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:51.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.488+0000 7fb1067fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:14:51.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.488+0000 7fb1077fe640 1 -- 172.21.15.7:0/1100780309 reap_dead start 2024-03-28T16:14:51.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.488+0000 7fb1067fc640 1 -- 172.21.15.7:0/1100780309 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1080a8c50 con 0x7fb0f00047c0 2024-03-28T16:14:51.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.488+0000 7fb1067fc640 10 monclient: _finish_auth 0 2024-03-28T16:14:51.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.488+0000 7fb1067fc640 10 monclient: _check_auth_tickets 2024-03-28T16:14:51.497 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.488+0000 7fb1067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:51.497 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.488+0000 7fb10d3dd640 5 monclient: authenticate success, global_id 4173 2024-03-28T16:14:51.497 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.488+0000 7fb10d3dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:14:51.498 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.490+0000 7fb1067fc640 1 --2- 172.21.15.7:0/1100780309 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0f00047c0 0x7fb0f0004c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc009a70 tx=0x7fb0f00057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc99e0fbe29f5f34 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:51.498 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.493+0000 7fb105ffb640 1 -- 172.21.15.7:0/1100780309 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc02c680 con 0x7fb0f00047c0 2024-03-28T16:14:51.498 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.493+0000 7fb105ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:51.498 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.494+0000 7fb105ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:14:51.498 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.494+0000 7fb105ffb640 10 monclient: dump: 2024-03-28T16:14:51.499 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:14:51.499 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:51.499 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:51.499 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:51.499 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:51.499 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:51.499 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:51.499 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:51.499 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:51.499 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:14:51.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.494+0000 7fb10d3dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:14:51.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.495+0000 7fb105ffb640 1 -- 172.21.15.7:0/1100780309 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0fc02c7e0 con 0x7fb0f00047c0 2024-03-28T16:14:51.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.495+0000 7fb105ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:51.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.495+0000 7fb10d3dd640 10 monclient: get_monmap_and_config success 2024-03-28T16:14:51.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.495+0000 7fb10d3dd640 4 set_mon_vals no callback set 2024-03-28T16:14:51.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.496+0000 7fb10d3dd640 10 monclient: shutdown 2024-03-28T16:14:51.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.496+0000 7fb10d3dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:51.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.496+0000 7fb10d3dd640 1 -- 172.21.15.7:0/1100780309 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0f00047c0 msgr2=0x7fb0f0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:51.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.496+0000 7fb10d3dd640 1 --2- 172.21.15.7:0/1100780309 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0f00047c0 0x7fb0f0004c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc009a70 tx=0x7fb0f00057f0 comp rx=0 tx=0).stop 2024-03-28T16:14:51.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.496+0000 7fb10d3dd640 1 -- 172.21.15.7:0/1100780309 shutdown_connections 2024-03-28T16:14:51.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.496+0000 7fb10d3dd640 1 --2- 172.21.15.7:0/1100780309 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0f00047c0 0x7fb0f0004c40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:51.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.496+0000 7fb10d3dd640 1 -- 172.21.15.7:0/1100780309 >> 172.21.15.7:0/1100780309 conn(0x7fb1080735d0 msgr2=0x7fb1080739e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:51.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.497+0000 7fb10d3dd640 1 -- 172.21.15.7:0/1100780309 shutdown_connections 2024-03-28T16:14:51.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.497+0000 7fb10d3dd640 1 -- 172.21.15.7:0/1100780309 wait complete. 2024-03-28T16:14:51.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.498+0000 7fb10d3dd640 10 monclient: build_initial_monmap 2024-03-28T16:14:51.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.498+0000 7fb10d3dd640 10 monclient: monmap: 2024-03-28T16:14:51.504 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-28T16:14:51.504 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:14:51.504 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-28T16:14:51.505 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-28T16:14:51.505 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:51.505 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:51.505 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:14:51.505 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:14:51.505 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:14:51.505 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:14:51.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.498+0000 7fb10d3dd640 1 librados: starting msgr at 2024-03-28T16:14:51.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.498+0000 7fb10d3dd640 1 librados: starting objecter 2024-03-28T16:14:51.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.499+0000 7fb10d3dd640 1 Processor -- start 2024-03-28T16:14:51.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.499+0000 7fb10d3dd640 1 -- start start 2024-03-28T16:14:51.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.499+0000 7fb10d3dd640 1 librados: setting wanted keys 2024-03-28T16:14:51.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.499+0000 7fb10d3dd640 1 librados: calling monclient init 2024-03-28T16:14:51.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.499+0000 7fb10d3dd640 10 monclient: init 2024-03-28T16:14:51.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.500+0000 7fb10d3dd640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:51.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.500+0000 7fb10d3dd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-28T16:14:51.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.500+0000 7fb10d3dd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0f00047c0 0x7fb1080aac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:51.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.500+0000 7fb10d3dd640 10 monclient(hunting): picked mon.noname-c con 0x7fb0f00047c0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:51.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.501+0000 7fb10d3dd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1081431d0 0x7fb108140870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:51.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.501+0000 7fb10d3dd640 10 monclient(hunting): picked mon.noname-a con 0x7fb1081431d0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:14:51.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.501+0000 7fb10d3dd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb108140db0 0x7fb1081d57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:51.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.501+0000 7fb10d3dd640 10 monclient(hunting): picked mon.noname-b con 0x7fb108140db0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:14:51.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.501+0000 7fb10d3dd640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:51.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.501+0000 7fb10d3dd640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1081436f0 con 0x7fb1081431d0 2024-03-28T16:14:51.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.501+0000 7fb10d3dd640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:51.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.501+0000 7fb10d3dd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb108143b80 con 0x7fb108140db0 2024-03-28T16:14:51.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.501+0000 7fb10d3dd640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:51.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.501+0000 7fb10d3dd640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1080ac490 con 0x7fb0f00047c0 2024-03-28T16:14:51.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.501+0000 7fb106ffd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0f00047c0 0x7fb1080aac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:51.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.501+0000 7fb10d3dd640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:51.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.501+0000 7fb10d3dd640 10 monclient(hunting): authenticate will time out at 1358.643641s 2024-03-28T16:14:51.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.502+0000 7fb1077fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb108140db0 0x7fb1081d57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:51.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.502+0000 7fb106ffd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0f00047c0 0x7fb1080aac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:32778/0 (socket says 172.21.15.7:32778) 2024-03-28T16:14:51.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.502+0000 7fb106ffd640 1 -- 172.21.15.7:0/3203702041 learned_addr learned my addr 172.21.15.7:0/3203702041 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:14:51.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.502+0000 7fb106ffd640 10 monclient(hunting): get_auth_request con 0x7fb0f00047c0 auth_method 0 2024-03-28T16:14:51.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.502+0000 7fb106ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:51.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.502+0000 7fb106ffd640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:51.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.502+0000 7fb106ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:51.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.502+0000 7fb1077fe640 10 monclient(hunting): get_auth_request con 0x7fb108140db0 auth_method 0 2024-03-28T16:14:51.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.502+0000 7fb1077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:51.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.502+0000 7fb1077fe640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:51.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.502+0000 7fb1077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:51.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.502+0000 7fb106ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:51.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.502+0000 7fb106ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:51.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.502+0000 7fb106ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:51.508 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.503+0000 7fb1077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:51.508 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.503+0000 7fb1077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:51.508 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.503+0000 7fb1077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:51.509 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.506+0000 7fb106ffd640 10 monclient(hunting): handle_auth_done global_id 4166 payload 824 2024-03-28T16:14:51.509 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.506+0000 7fb106ffd640 1 -- 172.21.15.7:0/3203702041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb108140db0 msgr2=0x7fb1081d57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:51.509 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.506+0000 7fb106ffd640 1 --2- 172.21.15.7:0/3203702041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb108140db0 0x7fb1081d57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:51.509 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.506+0000 7fb106ffd640 1 -- 172.21.15.7:0/3203702041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1081431d0 msgr2=0x7fb108140870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:51.509 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.506+0000 7fb106ffd640 1 --2- 172.21.15.7:0/3203702041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1081431d0 0x7fb108140870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:51.509 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.506+0000 7fb106ffd640 10 monclient: _finish_hunting 0 2024-03-28T16:14:51.509 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.506+0000 7fb106ffd640 1 monclient: found mon.noname-c 2024-03-28T16:14:51.509 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.507+0000 7fb106ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:51.509 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.507+0000 7fb106ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-03-28T16:14:51.510 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.507+0000 7fb106ffd640 1 -- 172.21.15.7:0/3203702041 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb108142600 con 0x7fb0f00047c0 2024-03-28T16:14:51.511 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.510+0000 7fb106ffd640 1 --2- 172.21.15.7:0/3203702041 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0f00047c0 0x7fb1080aac00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb0f800f770 tx=0x7fb0f800fc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:51.511 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.510+0000 7fb0ef7fe640 10 client.?.objecter ms_handle_connect 0x7fb0f00047c0 2024-03-28T16:14:51.511 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.510+0000 7fb0ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-28T16:14:51.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.516+0000 7fb0ef7fe640 1 -- 172.21.15.7:0/3203702041 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f800fe80 con 0x7fb0f00047c0 2024-03-28T16:14:51.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.516+0000 7fb0ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:51.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.516+0000 7fb0ef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:14:51.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.516+0000 7fb0ef7fe640 10 monclient: dump: 2024-03-28T16:14:51.519 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:14:51.519 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:51.519 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:51.519 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:51.519 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:51.519 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:51.519 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:51.519 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:51.519 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:51.520 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:14:51.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.516+0000 7fb0ef7fe640 10 monclient: _finish_auth 0 2024-03-28T16:14:51.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.516+0000 7fb0ef7fe640 10 monclient: _check_auth_tickets 2024-03-28T16:14:51.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.516+0000 7fb0ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:51.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.516+0000 7fb0ef7fe640 1 -- 172.21.15.7:0/3203702041 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f80050d0 con 0x7fb0f00047c0 2024-03-28T16:14:51.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.516+0000 7fb0ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:51.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.516+0000 7fb0ef7fe640 1 -- 172.21.15.7:0/3203702041 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f8005480 con 0x7fb0f00047c0 2024-03-28T16:14:51.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.516+0000 7fb0ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:51.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.516+0000 7fb0ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:14:51.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.516+0000 7fb0ef7fe640 10 monclient: dump: 2024-03-28T16:14:51.520 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:14:51.520 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:51.520 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:51.520 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:51.520 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:51.521 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:51.521 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:51.521 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:51.521 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:51.521 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:14:51.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.516+0000 7fb1057fa640 4 set_mon_vals no callback set 2024-03-28T16:14:51.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.516+0000 7fb10d3dd640 5 monclient: authenticate success, global_id 4166 2024-03-28T16:14:51.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.516+0000 7fb10d3dd640 10 monclient: _renew_subs 2024-03-28T16:14:51.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.517+0000 7fb10d3dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:14:51.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.517+0000 7fb10d3dd640 1 -- 172.21.15.7:0/3203702041 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb108141210 con 0x7fb0f00047c0 2024-03-28T16:14:51.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.517+0000 7fb10d3dd640 10 client.4166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:14:51.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.517+0000 7fb10d3dd640 10 monclient: _renew_subs 2024-03-28T16:14:51.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.517+0000 7fb10d3dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:14:51.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.517+0000 7fb10d3dd640 1 -- 172.21.15.7:0/3203702041 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1081d5d00 con 0x7fb0f00047c0 2024-03-28T16:14:51.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.517+0000 7fb10d3dd640 1 librados: init done 2024-03-28T16:14:51.522 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.518+0000 7fb10d3dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:51.522 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.518+0000 7fb10d3dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:51.522 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.518+0000 7fb10d3dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:14:51.522 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.519+0000 7fb10d3dd640 1 -- 172.21.15.7:0/3203702041 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1080670c0 con 0x7fb0f00047c0 2024-03-28T16:14:51.538 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.537+0000 7fb0ef7fe640 1 -- 172.21.15.7:0/3203702041 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 2) v1 ==== 44520+0+0 (secure 0 0 0) 0x7fb0f800d8c0 con 0x7fb0f00047c0 2024-03-28T16:14:51.541 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.540+0000 7fb0ef7fe640 1 -- 172.21.15.7:0/3203702041 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fb0f8014e30 con 0x7fb0f00047c0 2024-03-28T16:14:51.541 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.540+0000 7fb0ef7fe640 10 client.4166.objecter ms_dispatch 0x7fb108000b90 osd_map(10..10 src has 1..10) v4 2024-03-28T16:14:51.541 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.540+0000 7fb0ef7fe640 3 client.4166.objecter handle_osd_map got epochs [10,10] > 0 2024-03-28T16:14:51.542 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.540+0000 7fb0ef7fe640 3 client.4166.objecter handle_osd_map decoding full epoch 10 2024-03-28T16:14:51.542 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.541+0000 7fb0ef7fe640 20 client.4166.objecter dump_active .. 0 homeless 2024-03-28T16:14:51.693 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:51.692+0000 7efde1617040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T16:14:51.772 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.771+0000 7fb0ef7fe640 1 -- 172.21.15.7:0/3203702041 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fb0f8003b10 con 0x7fb0f00047c0 2024-03-28T16:14:51.773 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.771+0000 7fb0ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:51.773 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.771+0000 7fb0ef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:14:51.782 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:51.781+0000 7efde1617040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T16:14:51.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.896+0000 7fb10d3dd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:14:51.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.896+0000 7fb10d3dd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:14:51.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.896+0000 7fb10d3dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:14:51.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:51.896+0000 7fb10d3dd640 1 -- 172.21.15.7:0/3203702041 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1080700c0 con 0x7fb0f00047c0 2024-03-28T16:14:52.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:52.195+0000 7fb0ef7fe640 1 -- 172.21.15.7:0/3203702041 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mgrmap(e 3) v1 ==== 88848+0+0 (secure 0 0 0) 0x7fb0f80035f0 con 0x7fb0f00047c0 2024-03-28T16:14:52.225 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:52.224+0000 7fb0ef7fe640 1 -- 172.21.15.7:0/3203702041 <== mon.2 v2:172.21.15.7:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fb0f80579c0 con 0x7fb0f00047c0 2024-03-28T16:14:52.225 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:52.224+0000 7fb0ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:14:52.226 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:52.224+0000 7fb0ef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:14:52.226 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-28T16:14:52.226 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"808ff1a1-2b17-4c35-a89a-340dbbf674ee","created":"2024-03-28T16:14:44.591599+0000","modified":"2024-03-28T16:14:49.757700+0000","last_up_change":"0.000000","last_in_change":"2024-03-28T16:14:49.757700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a19c6856-78b9-4919-97ce-6940a5af982c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2afee73d-3369-4fd2-bfc5-36dee610daf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0473d84c-e864-4373-b9d1-283130278363","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b64fd5e7-cdec-4bd0-b5e2-dfc8d4f6b9ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8f7552db-5651-430f-a826-3c6843bffb99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2aabdbd2-d870-4d6d-823e-89b008ea3108","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fbbb7557-08d4-48d4-9b88-6bb1d680a87d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"805f54d9-03d5-4b23-aef3-e1d55bb09f87","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:14:52.227 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:52.225+0000 7fb10d3dd640 10 librados: watch_flush enter 2024-03-28T16:14:52.227 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:52.225+0000 7fb10d3dd640 10 librados: watch_flush exit 2024-03-28T16:14:52.227 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:52.225+0000 7fb10d3dd640 20 client.4166.objecter shutdown clearing up homeless session... 2024-03-28T16:14:52.227 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:52.225+0000 7fb10d3dd640 10 client.4166.objecter successfully canceled tick 2024-03-28T16:14:52.227 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:52.225+0000 7fb10d3dd640 10 monclient: shutdown 2024-03-28T16:14:52.227 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:52.225+0000 7fb10d3dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:52.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:52.225+0000 7fb10d3dd640 1 -- 172.21.15.7:0/3203702041 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0f00047c0 msgr2=0x7fb1080aac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:52.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:52.225+0000 7fb10d3dd640 1 --2- 172.21.15.7:0/3203702041 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0f00047c0 0x7fb1080aac00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb0f800f770 tx=0x7fb0f800fc30 comp rx=0 tx=0).stop 2024-03-28T16:14:52.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:52.226+0000 7fb10d3dd640 1 -- 172.21.15.7:0/3203702041 shutdown_connections 2024-03-28T16:14:52.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:52.226+0000 7fb10d3dd640 1 --2- 172.21.15.7:0/3203702041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1081431d0 0x7fb108140870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:52.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:52.226+0000 7fb10d3dd640 1 --2- 172.21.15.7:0/3203702041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb108140db0 0x7fb1081d57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:52.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:52.226+0000 7fb10d3dd640 1 --2- 172.21.15.7:0/3203702041 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb0f00047c0 0x7fb1080aac00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:52.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:52.226+0000 7fb10d3dd640 1 -- 172.21.15.7:0/3203702041 >> 172.21.15.7:0/3203702041 conn(0x7fb1080735d0 msgr2=0x7fb1080b6f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:52.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:52.226+0000 7fb10d3dd640 1 -- 172.21.15.7:0/3203702041 shutdown_connections 2024-03-28T16:14:52.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:52.226+0000 7fb10d3dd640 1 -- 172.21.15.7:0/3203702041 wait complete. 2024-03-28T16:14:52.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:52.226+0000 7fb10d3dd640 1 librados: shutdown 2024-03-28T16:14:52.244 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-28T16:14:52.476 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:52.475+0000 7efde1617040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T16:14:52.554 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:52.554+0000 7efde1617040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T16:14:52.627 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:52.626+0000 7efde1617040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T16:14:52.715 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:52.714+0000 7efde1617040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T16:14:52.784 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:52.783+0000 7efde1617040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T16:14:52.984 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:52.984+0000 7efde1617040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T16:14:53.072 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:53.071+0000 7efde1617040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T16:14:53.140 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:53.139+0000 7efde1617040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T16:14:53.220 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:53.219+0000 7efde1617040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T16:14:53.293 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:53.292+0000 7efde1617040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T16:14:53.849 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:53.848+0000 7efde1617040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T16:14:54.075 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:54.075+0000 7efde1617040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T16:14:54.235 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:54.234+0000 7efde1617040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T16:14:54.309 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:54.308+0000 7efde1617040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T16:14:54.510 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:54.509+0000 7efde1617040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T16:14:54.669 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:54.668+0000 7efde1617040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T16:14:54.942 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:54.941+0000 7efde1617040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T16:14:55.014 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:55.013+0000 7efde1617040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T16:14:55.233 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:55.232+0000 7efde1617040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T16:14:56.197 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:56.196+0000 7efde1617040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T16:14:56.521 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:56.520+0000 7efde1617040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T16:14:56.572 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:56.571+0000 7efde1617040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T16:14:56.696 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:56.695+0000 7efde1617040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T16:14:57.155 INFO:tasks.ceph.osd.7.smithi083.stderr:==00:00:00:07.254 35295== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:57.216 INFO:tasks.ceph.osd.7.smithi083.stderr:==00:00:00:07.315 35295== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:57.263 INFO:tasks.ceph.osd.6.smithi083.stderr:==00:00:00:07.348 35297== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:57.326 INFO:tasks.ceph.osd.6.smithi083.stderr:==00:00:00:07.411 35297== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:57.495 INFO:tasks.ceph.osd.4.smithi083.stderr:==00:00:00:07.605 35294== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:57.514 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:07.624 34919== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:57.556 INFO:tasks.ceph.osd.4.smithi083.stderr:==00:00:00:07.665 35294== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:57.577 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:07.686 34919== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:57.695 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:07.810 34918== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:57.767 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:14:57.761+0000 5dc2480 -1 Falling back to public interface 2024-03-28T16:14:57.784 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:07.899 34918== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:57.799 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:07.890 34921== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:57.863 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:07.953 34921== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:57.895 INFO:tasks.ceph.osd.6.smithi083.stderr:2024-03-28T16:14:57.886+0000 5dc2480 -1 Falling back to public interface 2024-03-28T16:14:57.924 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:08.018 34920== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:57.988 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:08.082 34920== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:58.125 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:14:58.117+0000 5dc2480 -1 Falling back to public interface 2024-03-28T16:14:58.172 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-03-28T16:14:58.164+0000 5dc2480 -1 Falling back to public interface 2024-03-28T16:14:58.257 INFO:tasks.ceph.osd.7.smithi083.stderr:==00:00:00:08.357 35295== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:58.327 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-03-28T16:14:58.320+0000 5dc2480 -1 Falling back to public interface 2024-03-28T16:14:58.363 INFO:tasks.ceph.osd.6.smithi083.stderr:==00:00:00:08.447 35297== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:58.437 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:14:58.427+0000 5dc2480 -1 Falling back to public interface 2024-03-28T16:14:58.534 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:58.533+0000 7efd82087640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-28T16:14:58.570 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:14:58.561+0000 5dc2480 -1 Falling back to public interface 2024-03-28T16:14:58.622 INFO:tasks.ceph.osd.7.smithi083.stderr:==00:00:00:08.721 35295== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:58.655 INFO:tasks.ceph.osd.4.smithi083.stderr:==00:00:00:08.765 35294== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:58.689 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:08.799 34919== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:58.749 INFO:tasks.ceph.osd.6.smithi083.stderr:==00:00:00:08.833 35297== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:58.767 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:14:58.766+0000 7efd82087640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-28T16:14:58.821 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:08.935 34918== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:58.936 INFO:tasks.ceph.osd.7.smithi083.stderr:==00:00:00:09.036 35295== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:58.958 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:09.048 34921== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.025 INFO:tasks.ceph.osd.4.smithi083.stderr:==00:00:00:09.135 35294== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.047 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:09.156 34919== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.048 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-28T16:14:59.085 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:09.179 34920== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.114 INFO:tasks.ceph.osd.6.smithi083.stderr:==00:00:00:09.199 35297== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.145 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.144+0000 7f2ac76bd640 10 monclient: get_monmap_and_config 2024-03-28T16:14:59.146 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.144+0000 7f2ac76bd640 10 monclient: build_initial_monmap 2024-03-28T16:14:59.146 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.144+0000 7f2ac76bd640 10 monclient: monmap: 2024-03-28T16:14:59.146 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-28T16:14:59.146 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:59.146 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:14:59.146481+0000 2024-03-28T16:14:59.146 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:14:59.146481+0000 2024-03-28T16:14:59.146 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:59.146 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:59.146 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:14:59.146 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:14:59.146 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:14:59.146 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:14:59.146 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:14:59.146 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:14:59.149 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.145+0000 7f2ac76bd640 1 Processor -- start 2024-03-28T16:14:59.149 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.145+0000 7f2ac76bd640 1 -- start start 2024-03-28T16:14:59.149 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.145+0000 7f2ac76bd640 10 monclient: init 2024-03-28T16:14:59.149 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.146+0000 7f2ac76bd640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:59.149 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.146+0000 7f2ac76bd640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-03-28T16:14:59.149 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.146+0000 7f2ac76bd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2ac006a5b0 0x7f2ac00af700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:59.149 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.146+0000 7f2ac76bd640 10 monclient(hunting): picked mon.noname-b con 0x7f2ac006a5b0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:59.149 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.146+0000 7f2ac76bd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2ac00afc40 0x7f2ac0144f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:59.149 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.146+0000 7f2ac76bd640 10 monclient(hunting): picked mon.noname-c con 0x7f2ac00afc40 addr v2:172.21.15.83:3300/0 2024-03-28T16:14:59.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.146+0000 7f2ac76bd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ac0145480 addr v1:172.21.15.7:6789/0 2024-03-28T16:14:59.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.146+0000 7f2ac76bd640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac00acfb0 con 0x7f2ac0145480 2024-03-28T16:14:59.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.146+0000 7f2ac76bd640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:59.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.146+0000 7f2ac76bd640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac00b00a0 con 0x7f2ac00afc40 2024-03-28T16:14:59.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.146+0000 7f2ac76bd640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:59.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.146+0000 7f2ac76bd640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac006b350 con 0x7f2ac006a5b0 2024-03-28T16:14:59.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.146+0000 7f2ac76bd640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:59.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.146+0000 7f2ac76bd640 10 monclient(hunting): authenticate will time out at 1366.288139s 2024-03-28T16:14:59.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.146+0000 7f2ac5459640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2ac006a5b0 0x7f2ac00af700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:59.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.146+0000 7f2ac4c58640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2ac00afc40 0x7f2ac0144f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:59.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.146+0000 7f2ac5c5a640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2ac0145480 0x7f2ac00ad920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:52626/0 (socket says 172.21.15.7:52626) 2024-03-28T16:14:59.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.146+0000 7f2ac5c5a640 1 -- 172.21.15.7:0/1948795571 learned_addr learned my addr 172.21.15.7:0/1948795571 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-28T16:14:59.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.147+0000 7f2ac4c58640 10 monclient(hunting): get_auth_request con 0x7f2ac00afc40 auth_method 0 2024-03-28T16:14:59.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.147+0000 7f2ac4c58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:59.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.147+0000 7f2ac4c58640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:59.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.147+0000 7f2ac4c58640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:59.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.147+0000 7f2ac5459640 10 monclient(hunting): get_auth_request con 0x7f2ac006a5b0 auth_method 0 2024-03-28T16:14:59.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.147+0000 7f2ac5459640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:59.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.147+0000 7f2ac5459640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:59.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.147+0000 7f2ac5459640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:59.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.147+0000 7f2ac4c58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:59.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.147+0000 7f2ac4c58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:59.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.147+0000 7f2ac4c58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:59.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.148+0000 7f2ac5459640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:59.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.148+0000 7f2ac5459640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:59.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.148+0000 7f2ac5459640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:59.152 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.149+0000 7f2ac4c58640 10 monclient(hunting): handle_auth_done global_id 4210 payload 274 2024-03-28T16:14:59.152 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.149+0000 7f2ac4c58640 1 -- 172.21.15.7:0/1948795571 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2ac006a5b0 msgr2=0x7f2ac00af700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:14:59.152 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.149+0000 7f2ac4c58640 1 --2- 172.21.15.7:0/1948795571 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2ac006a5b0 0x7f2ac00af700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:59.152 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.149+0000 7f2ac4c58640 1 -- 172.21.15.7:0/1948795571 >> v1:172.21.15.7:6789/0 conn(0x7f2ac0145480 legacy=0x7f2ac00ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:59.152 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.149+0000 7f2ac4c58640 10 monclient: _finish_hunting 0 2024-03-28T16:14:59.152 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.149+0000 7f2ac4c58640 1 monclient: found mon.noname-c 2024-03-28T16:14:59.152 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.149+0000 7f2ac4c58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:59.153 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.149+0000 7f2ac4c58640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3300/0 2024-03-28T16:14:59.153 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.149+0000 7f2ac4c58640 1 -- 172.21.15.7:0/1948795571 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac006a990 con 0x7f2ac00afc40 2024-03-28T16:14:59.153 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.151+0000 7f2ac4c58640 1 --2- 172.21.15.7:0/1948795571 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ac00afc40 0x7f2ac0144f40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2ab00097a0 tx=0x7f2ab002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d42c6b9a0fea52a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:59.157 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ab7fff640 1 -- 172.21.15.7:0/1948795571 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab0031830 con 0x7f2ac00afc40 2024-03-28T16:14:59.157 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ab7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:59.157 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ab7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:14:59.157 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ab7fff640 10 monclient: dump: 2024-03-28T16:14:59.157 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:14:59.157 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:59.157 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:59.157 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:59.157 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:59.157 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:59.157 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:59.157 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:59.158 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:59.158 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:14:59.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ab7fff640 10 monclient: _finish_auth 0 2024-03-28T16:14:59.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ab7fff640 10 monclient: _check_auth_tickets 2024-03-28T16:14:59.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ab7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:59.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ab7fff640 1 -- 172.21.15.7:0/1948795571 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab0031990 con 0x7f2ac00afc40 2024-03-28T16:14:59.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ab7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:59.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ab7fff640 1 -- 172.21.15.7:0/1948795571 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab0031d20 con 0x7f2ac00afc40 2024-03-28T16:14:59.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ac76bd640 5 monclient: authenticate success, global_id 4210 2024-03-28T16:14:59.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ac76bd640 10 monclient: get_monmap_and_config success 2024-03-28T16:14:59.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ac76bd640 4 set_mon_vals no callback set 2024-03-28T16:14:59.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ab7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:59.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ab7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:14:59.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ab7fff640 10 monclient: dump: 2024-03-28T16:14:59.159 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:14:59.159 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:59.159 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:59.159 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:59.159 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:59.159 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:59.159 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:59.159 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:59.159 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:59.159 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:14:59.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ac76bd640 10 monclient: shutdown 2024-03-28T16:14:59.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ac76bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:59.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ac76bd640 1 -- 172.21.15.7:0/1948795571 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ac00afc40 msgr2=0x7f2ac0144f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:59.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ac76bd640 1 --2- 172.21.15.7:0/1948795571 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ac00afc40 0x7f2ac0144f40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2ab00097a0 tx=0x7f2ab002b8d0 comp rx=0 tx=0).stop 2024-03-28T16:14:59.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ac76bd640 1 -- 172.21.15.7:0/1948795571 shutdown_connections 2024-03-28T16:14:59.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.155+0000 7f2ac76bd640 1 --2- 172.21.15.7:0/1948795571 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ac00afc40 0x7f2ac0144f40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:59.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.156+0000 7f2ac76bd640 1 --2- 172.21.15.7:0/1948795571 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2ac006a5b0 0x7f2ac00af700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:59.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.156+0000 7f2ac76bd640 1 -- 172.21.15.7:0/1948795571 >> 172.21.15.7:0/1948795571 conn(0x7f2ac00b4ee0 msgr2=0x7f2ac00b52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:59.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.156+0000 7f2ac76bd640 1 -- 172.21.15.7:0/1948795571 shutdown_connections 2024-03-28T16:14:59.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.156+0000 7f2ac76bd640 1 -- 172.21.15.7:0/1948795571 wait complete. 2024-03-28T16:14:59.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.156+0000 7f2ac76bd640 10 monclient: build_initial_monmap 2024-03-28T16:14:59.164 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.156+0000 7f2ac76bd640 10 monclient: monmap: 2024-03-28T16:14:59.164 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-28T16:14:59.164 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:14:59.164 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-28T16:14:59.164 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-28T16:14:59.164 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:14:59.164 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:59.164 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:14:59.164 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:14:59.164 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:14:59.164 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:14:59.164 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 1 librados: starting msgr at 2024-03-28T16:14:59.165 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 1 librados: starting objecter 2024-03-28T16:14:59.165 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 1 Processor -- start 2024-03-28T16:14:59.165 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 1 -- start start 2024-03-28T16:14:59.165 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 1 librados: setting wanted keys 2024-03-28T16:14:59.165 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 1 librados: calling monclient init 2024-03-28T16:14:59.165 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 10 monclient: init 2024-03-28T16:14:59.165 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 10 monclient: _reopen_session rank -1 2024-03-28T16:14:59.165 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-28T16:14:59.165 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ac006a5b0 0x7f2ac01d9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:59.165 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 10 monclient(hunting): picked mon.noname-b con 0x7f2ac006a5b0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:14:59.165 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ac00afc40 0x7f2ac013f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:59.165 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 10 monclient(hunting): picked mon.noname-a con 0x7f2ac00afc40 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:14:59.165 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2ac0145480 0x7f2ac011c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:59.165 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 10 monclient(hunting): picked mon.noname-c con 0x7f2ac0145480 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:14:59.165 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:59.166 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac013fdf0 con 0x7f2ac00afc40 2024-03-28T16:14:59.166 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:59.166 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac01cdb00 con 0x7f2ac006a5b0 2024-03-28T16:14:59.166 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 10 monclient(hunting): start opening mon connection 2024-03-28T16:14:59.166 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac01da530 con 0x7f2ac0145480 2024-03-28T16:14:59.166 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 10 monclient(hunting): _renew_subs 2024-03-28T16:14:59.166 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.157+0000 7f2ac76bd640 10 monclient(hunting): authenticate will time out at 1366.299650s 2024-03-28T16:14:59.166 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.158+0000 7f2ac4c58640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ac00afc40 0x7f2ac013f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:59.166 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.158+0000 7f2ac5c5a640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2ac0145480 0x7f2ac011c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:59.166 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.158+0000 7f2ac5459640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ac006a5b0 0x7f2ac01d9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:59.166 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.158+0000 7f2ac5459640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ac006a5b0 0x7f2ac01d9ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:40692/0 (socket says 172.21.15.7:40692) 2024-03-28T16:14:59.166 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.158+0000 7f2ac5459640 1 -- 172.21.15.7:0/2542619692 learned_addr learned my addr 172.21.15.7:0/2542619692 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:14:59.166 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.158+0000 7f2ac5459640 10 monclient(hunting): get_auth_request con 0x7f2ac006a5b0 auth_method 0 2024-03-28T16:14:59.166 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.158+0000 7f2ac5459640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:59.166 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.158+0000 7f2ac5459640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:59.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.158+0000 7f2ac5459640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:59.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.158+0000 7f2ac4c58640 10 monclient(hunting): get_auth_request con 0x7f2ac00afc40 auth_method 0 2024-03-28T16:14:59.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.158+0000 7f2ac4c58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:59.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.158+0000 7f2ac4c58640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:59.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.158+0000 7f2ac4c58640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:59.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.158+0000 7f2ac5c5a640 10 monclient(hunting): get_auth_request con 0x7f2ac0145480 auth_method 0 2024-03-28T16:14:59.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.158+0000 7f2ac5c5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:14:59.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.158+0000 7f2ac5c5a640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:14:59.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.158+0000 7f2ac5c5a640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:14:59.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.159+0000 7f2ac5459640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:59.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.159+0000 7f2ac5459640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:59.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.159+0000 7f2ac5459640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:59.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.159+0000 7f2ac4c58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:59.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.159+0000 7f2ac4c58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:59.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.159+0000 7f2ac4c58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:59.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.159+0000 7f2ac5c5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:14:59.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.159+0000 7f2ac5c5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:14:59.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.159+0000 7f2ac5c5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:14:59.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.161+0000 7f2ac5459640 10 monclient(hunting): handle_auth_done global_id 4213 payload 824 2024-03-28T16:14:59.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.161+0000 7f2ac5459640 1 -- 172.21.15.7:0/2542619692 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2ac0145480 msgr2=0x7f2ac011c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:59.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.161+0000 7f2ac5459640 1 --2- 172.21.15.7:0/2542619692 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2ac0145480 0x7f2ac011c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:59.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.161+0000 7f2ac5459640 1 -- 172.21.15.7:0/2542619692 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ac00afc40 msgr2=0x7f2ac013f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:59.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.161+0000 7f2ac5459640 1 --2- 172.21.15.7:0/2542619692 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ac00afc40 0x7f2ac013f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:59.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.161+0000 7f2ac5459640 10 monclient: _finish_hunting 0 2024-03-28T16:14:59.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.161+0000 7f2ac5459640 1 monclient: found mon.noname-b 2024-03-28T16:14:59.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.161+0000 7f2ac5459640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:14:59.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.161+0000 7f2ac5459640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:14:59.169 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.161+0000 7f2ac5459640 1 -- 172.21.15.7:0/2542619692 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab0009400 con 0x7f2ac006a5b0 2024-03-28T16:14:59.170 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.164+0000 7f2ac5459640 1 --2- 172.21.15.7:0/2542619692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ac006a5b0 0x7f2ac01d9ac0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00e9b0 tx=0x7f2abc00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:59.170 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.164+0000 7f2ab5ffb640 10 client.?.objecter ms_handle_connect 0x7f2ac006a5b0 2024-03-28T16:14:59.170 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.164+0000 7f2ab5ffb640 10 client.?.objecter resend_mon_ops 2024-03-28T16:14:59.170 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ab5ffb640 1 -- 172.21.15.7:0/2542619692 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc009670 con 0x7f2ac006a5b0 2024-03-28T16:14:59.170 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ab5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:59.170 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ab5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:14:59.170 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ab5ffb640 10 monclient: dump: 2024-03-28T16:14:59.170 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:14:59.170 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:59.170 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:59.171 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:59.171 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:59.171 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:59.171 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:59.171 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:59.171 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:59.171 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:14:59.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ab5ffb640 10 monclient: _finish_auth 0 2024-03-28T16:14:59.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ab5ffb640 10 monclient: _check_auth_tickets 2024-03-28T16:14:59.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ab5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:14:59.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ab5ffb640 1 -- 172.21.15.7:0/2542619692 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2abc00c9a0 con 0x7f2ac006a5b0 2024-03-28T16:14:59.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ab5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:14:59.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ab5ffb640 1 -- 172.21.15.7:0/2542619692 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc00cd50 con 0x7f2ac006a5b0 2024-03-28T16:14:59.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ab5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:14:59.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ab5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:14:59.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ab5ffb640 10 monclient: dump: 2024-03-28T16:14:59.172 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:14:59.172 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:14:59.172 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:59.172 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:14:59.172 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:14:59.172 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:14:59.172 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:14:59.172 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:14:59.172 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:14:59.172 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:14:59.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ac76bd640 5 monclient: authenticate success, global_id 4213 2024-03-28T16:14:59.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ac76bd640 10 monclient: _renew_subs 2024-03-28T16:14:59.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ac76bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:59.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ab6ffd640 4 set_mon_vals no callback set 2024-03-28T16:14:59.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ac76bd640 1 -- 172.21.15.7:0/2542619692 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac01da7e0 con 0x7f2ac006a5b0 2024-03-28T16:14:59.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ac76bd640 10 client.4213.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:14:59.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ac76bd640 10 monclient: _renew_subs 2024-03-28T16:14:59.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ac76bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:59.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ac76bd640 1 -- 172.21.15.7:0/2542619692 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac0140460 con 0x7f2ac006a5b0 2024-03-28T16:14:59.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.168+0000 7f2ac76bd640 1 librados: init done 2024-03-28T16:14:59.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.170+0000 7f2ac76bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:59.174 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.170+0000 7f2ac76bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:59.174 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.170+0000 7f2ac76bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:59.174 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.170+0000 7f2ac76bd640 1 -- 172.21.15.7:0/2542619692 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac0069430 con 0x7f2ac006a5b0 2024-03-28T16:14:59.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.193+0000 7f2ab5ffb640 1 -- 172.21.15.7:0/2542619692 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 88975+0+0 (secure 0 0 0) 0x7f2abc013470 con 0x7f2ac006a5b0 2024-03-28T16:14:59.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.193+0000 7f2ab5ffb640 1 --2- 172.21.15.7:0/2542619692 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f2aa406c410 0x7f2aa406e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:14:59.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.194+0000 7f2ac4c58640 1 --2- 172.21.15.7:0/2542619692 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f2aa406c410 0x7f2aa406e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:14:59.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.194+0000 7f2ac4c58640 10 monclient: get_auth_request con 0x7f2aa406c410 auth_method 0 2024-03-28T16:14:59.195 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:09.309 34918== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.194+0000 7f2ab5ffb640 10 client.4213.objecter ms_handle_connect 0x7f2aa406c410 2024-03-28T16:14:59.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.194+0000 7f2ac4c58640 1 --2- 172.21.15.7:0/2542619692 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f2aa406c410 0x7f2aa406e8d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0036000 tx=0x7f2ab003c040 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:14:59.208 INFO:tasks.ceph.osd.7.smithi083.stderr:==00:00:00:09.307 35295== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.317 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:09.405 34921== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.349 INFO:tasks.ceph.osd.4.smithi083.stderr:==00:00:00:09.459 35294== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.369 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:09.478 34919== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.404 INFO:tasks.ceph.osd.6.smithi083.stderr:==00:00:00:09.488 35297== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.439 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.438+0000 7f2ab5ffb640 1 -- 172.21.15.7:0/2542619692 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f2abc0589f0 con 0x7f2ac006a5b0 2024-03-28T16:14:59.439 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.438+0000 7f2ab5ffb640 10 client.4213.objecter ms_dispatch 0x7f2ac0000b90 osd_map(11..11 src has 1..11) v4 2024-03-28T16:14:59.439 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.438+0000 7f2ab5ffb640 3 client.4213.objecter handle_osd_map got epochs [11,11] > 0 2024-03-28T16:14:59.439 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.438+0000 7f2ab5ffb640 3 client.4213.objecter handle_osd_map decoding full epoch 11 2024-03-28T16:14:59.439 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.438+0000 7f2ab5ffb640 20 client.4213.objecter dump_active .. 0 homeless 2024-03-28T16:14:59.440 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.438+0000 7f2ab5ffb640 1 -- 172.21.15.7:0/2542619692 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2abc054d30 con 0x7f2ac006a5b0 2024-03-28T16:14:59.440 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.438+0000 7f2ab5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:14:59.440 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.438+0000 7f2ab5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:14:59.445 INFO:tasks.ceph.osd.5.smithi083.stderr:==00:00:00:09.528 35296== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.457 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:09.551 34920== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.487 INFO:tasks.ceph.osd.7.smithi083.stderr:==00:00:00:09.586 35295== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.506 INFO:tasks.ceph.osd.5.smithi083.stderr:==00:00:00:09.589 35296== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.511 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:09.625 34918== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.615 INFO:tasks.ceph.osd.4.smithi083.stderr:==00:00:00:09.725 35294== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.628 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.627+0000 7f2ac76bd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:14:59.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.627+0000 7f2ac76bd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:14:59.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.627+0000 7f2ac76bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:14:59.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.627+0000 7f2ac76bd640 1 -- 172.21.15.7:0/2542619692 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ac00ad280 con 0x7f2ac006a5b0 2024-03-28T16:14:59.629 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:09.720 34921== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.633 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:09.743 34919== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.708 INFO:tasks.ceph.osd.6.smithi083.stderr:==00:00:00:09.792 35297== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.752 INFO:tasks.ceph.osd.7.smithi083.stderr:==00:00:00:09.852 35295== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.780 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:09.874 34920== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.784 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:09.898 34918== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.814 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.813+0000 7f2ab5ffb640 1 -- 172.21.15.7:0/2542619692 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mgrmap(e 6) v1 ==== 89081+0+0 (secure 0 0 0) 0x7f2abc055100 con 0x7f2ac006a5b0 2024-03-28T16:14:59.816 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.815+0000 7f2ab5ffb640 1 -- 172.21.15.7:0/2542619692 <== mon.1 v2:172.21.15.83:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f2abc0760b0 con 0x7f2ac006a5b0 2024-03-28T16:14:59.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.816+0000 7f2ab5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:14:59.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.816+0000 7f2ab5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:14:59.817 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-28T16:14:59.817 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":11,"fsid":"808ff1a1-2b17-4c35-a89a-340dbbf674ee","created":"2024-03-28T16:14:44.591599+0000","modified":"2024-03-28T16:14:56.773245+0000","last_up_change":"0.000000","last_in_change":"2024-03-28T16:14:49.757700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a19c6856-78b9-4919-97ce-6940a5af982c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2afee73d-3369-4fd2-bfc5-36dee610daf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0473d84c-e864-4373-b9d1-283130278363","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b64fd5e7-cdec-4bd0-b5e2-dfc8d4f6b9ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8f7552db-5651-430f-a826-3c6843bffb99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2aabdbd2-d870-4d6d-823e-89b008ea3108","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fbbb7557-08d4-48d4-9b88-6bb1d680a87d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"805f54d9-03d5-4b23-aef3-e1d55bb09f87","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:14:59.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.818+0000 7f2ac76bd640 10 librados: watch_flush enter 2024-03-28T16:14:59.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.818+0000 7f2ac76bd640 10 librados: watch_flush exit 2024-03-28T16:14:59.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.818+0000 7f2ac76bd640 20 client.4213.objecter shutdown clearing up homeless session... 2024-03-28T16:14:59.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.818+0000 7f2ac76bd640 10 client.4213.objecter successfully canceled tick 2024-03-28T16:14:59.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.818+0000 7f2ac76bd640 1 -- 172.21.15.7:0/2542619692 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f2aa406c410 msgr2=0x7f2aa406e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:59.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.818+0000 7f2ac76bd640 1 --2- 172.21.15.7:0/2542619692 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f2aa406c410 0x7f2aa406e8d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0036000 tx=0x7f2ab003c040 comp rx=0 tx=0).stop 2024-03-28T16:14:59.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.818+0000 7f2ac76bd640 10 monclient: shutdown 2024-03-28T16:14:59.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.818+0000 7f2ac76bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:14:59.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.818+0000 7f2ac76bd640 1 -- 172.21.15.7:0/2542619692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ac006a5b0 msgr2=0x7f2ac01d9ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:14:59.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.818+0000 7f2ac76bd640 1 --2- 172.21.15.7:0/2542619692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ac006a5b0 0x7f2ac01d9ac0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00e9b0 tx=0x7f2abc00ee70 comp rx=0 tx=0).stop 2024-03-28T16:14:59.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.818+0000 7f2ac76bd640 1 -- 172.21.15.7:0/2542619692 shutdown_connections 2024-03-28T16:14:59.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.818+0000 7f2ac76bd640 1 --2- 172.21.15.7:0/2542619692 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f2aa406c410 0x7f2aa406e8d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:59.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.818+0000 7f2ac76bd640 1 --2- 172.21.15.7:0/2542619692 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ac00afc40 0x7f2ac013f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:59.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.818+0000 7f2ac76bd640 1 --2- 172.21.15.7:0/2542619692 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2ac0145480 0x7f2ac011c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:59.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.818+0000 7f2ac76bd640 1 --2- 172.21.15.7:0/2542619692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ac006a5b0 0x7f2ac01d9ac0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:14:59.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.818+0000 7f2ac76bd640 1 -- 172.21.15.7:0/2542619692 >> 172.21.15.7:0/2542619692 conn(0x7f2ac00b4ee0 msgr2=0x7f2ac0143800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:14:59.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.818+0000 7f2ac76bd640 1 -- 172.21.15.7:0/2542619692 shutdown_connections 2024-03-28T16:14:59.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.818+0000 7f2ac76bd640 1 -- 172.21.15.7:0/2542619692 wait complete. 2024-03-28T16:14:59.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:14:59.818+0000 7f2ac76bd640 1 librados: shutdown 2024-03-28T16:14:59.829 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-28T16:14:59.879 INFO:tasks.ceph.osd.4.smithi083.stderr:==00:00:00:09.989 35294== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.897 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:09.988 34921== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.901 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:10.011 34919== Warning: unimplemented fcntl command: 1038 2024-03-28T16:14:59.996 INFO:tasks.ceph.osd.6.smithi083.stderr:==00:00:00:10.080 35297== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.022 INFO:tasks.ceph.osd.7.smithi083.stderr:==00:00:00:10.121 35295== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.051 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:10.146 34920== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.059 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:10.173 34918== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.077 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:15:00.071+0000 5dc2480 -1 Falling back to public interface 2024-03-28T16:15:00.157 INFO:tasks.ceph.osd.4.smithi083.stderr:==00:00:00:10.267 35294== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.175 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:10.266 34921== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.176 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:10.285 34919== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.290 INFO:tasks.ceph.osd.6.smithi083.stderr:==00:00:00:10.375 35297== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.294 INFO:tasks.ceph.osd.7.smithi083.stderr:==00:00:00:10.393 35295== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.323 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:10.417 34920== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.326 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:10.440 34918== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.427 INFO:tasks.ceph.osd.4.smithi083.stderr:==00:00:00:10.537 35294== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.440 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:10.550 34919== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.441 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:10.531 34921== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.529 INFO:tasks.ceph.osd.5.smithi083.stderr:==00:00:00:10.613 35296== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.566 INFO:tasks.ceph.osd.7.smithi083.stderr:==00:00:00:10.665 35295== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.585 INFO:tasks.ceph.osd.6.smithi083.stderr:==00:00:00:10.669 35297== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.589 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:10.684 34920== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.595 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:10.709 34918== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.696 INFO:tasks.ceph.osd.4.smithi083.stderr:==00:00:00:10.806 35294== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.709 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:10.819 34919== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.713 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:10.804 34921== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.858 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:10.952 34920== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.863 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:10.977 34918== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.881 INFO:tasks.ceph.osd.6.smithi083.stderr:==00:00:00:10.965 35297== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.885 INFO:tasks.ceph.osd.5.smithi083.stderr:==00:00:00:10.968 35296== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.960 INFO:tasks.ceph.osd.4.smithi083.stderr:==00:00:00:11.070 35294== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.985 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:11.095 34919== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:00.993 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:11.084 34921== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.037 INFO:tasks.ceph.osd.7.smithi083.stderr:==00:00:00:11.137 35295== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.089 INFO:tasks.ceph.osd.7.smithi083.stderr:==00:00:00:11.189 35295== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.129 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:11.223 34920== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.133 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:11.247 34918== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.259 INFO:tasks.ceph.osd.5.smithi083.stderr:==00:00:00:11.342 35296== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.270 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:11.361 34921== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.400 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:11.494 34920== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.404 INFO:tasks.ceph.osd.6.smithi083.stderr:==00:00:00:11.489 35297== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.444 INFO:tasks.ceph.osd.4.smithi083.stderr:==00:00:00:11.554 35294== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.465 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:11.575 34919== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.480 INFO:tasks.ceph.osd.6.smithi083.stderr:==00:00:00:11.564 35297== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.515 INFO:tasks.ceph.osd.4.smithi083.stderr:==00:00:00:11.625 35294== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.521 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:11.631 34919== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.569 INFO:tasks.ceph.osd.5.smithi083.stderr:==00:00:00:11.652 35296== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.612 INFO:tasks.ceph.osd.5.smithi083.stderr:==00:00:00:11.695 35296== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.644 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:11.759 34918== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.653 INFO:tasks.ceph.osd.5.smithi083.stderr:==00:00:00:11.737 35296== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.720 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:11.834 34918== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.806 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:11.896 34921== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.862 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:11.952 34921== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.890 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:11.984 34920== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.942 INFO:tasks.ceph.osd.5.smithi083.stderr:==00:00:00:12.025 35296== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:01.946 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:12.040 34920== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:02.231 INFO:tasks.ceph.osd.5.smithi083.stderr:==00:00:00:12.314 35296== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:02.286 INFO:tasks.ceph.osd.5.smithi083.stderr:==00:00:00:12.369 35296== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:02.586 INFO:tasks.ceph.osd.5.smithi083.stderr:==00:00:00:12.669 35296== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:02.658 INFO:tasks.ceph.osd.5.smithi083.stderr:==00:00:00:12.741 35296== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:06.636 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-28T16:15:06.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.726+0000 7f35bc5b1640 10 monclient: get_monmap_and_config 2024-03-28T16:15:06.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.726+0000 7f35bc5b1640 10 monclient: build_initial_monmap 2024-03-28T16:15:06.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.726+0000 7f35bc5b1640 10 monclient: monmap: 2024-03-28T16:15:06.728 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-28T16:15:06.728 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:06.728 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:15:06.728484+0000 2024-03-28T16:15:06.728 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:15:06.728484+0000 2024-03-28T16:15:06.728 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:06.728 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:06.728 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:06.728 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:06.729 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:06.729 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:06.729 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:06.729 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:06.731 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.727+0000 7f35bc5b1640 1 Processor -- start 2024-03-28T16:15:06.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.727+0000 7f35bc5b1640 1 -- start start 2024-03-28T16:15:06.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.727+0000 7f35bc5b1640 10 monclient: init 2024-03-28T16:15:06.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.728+0000 7f35bc5b1640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:06.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.728+0000 7f35bc5b1640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-03-28T16:15:06.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.728+0000 7f35bc5b1640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f35b40719a0 0x7f35b413e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:06.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.728+0000 7f35bc5b1640 10 monclient(hunting): picked mon.noname-a con 0x7f35b40719a0 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:06.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.728+0000 7f35bc5b1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35b413ece0 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:06.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.728+0000 7f35bc5b1640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f35b4141e50 0x7f35b414be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:06.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.728+0000 7f35bc5b1640 10 monclient(hunting): picked mon.noname-b con 0x7f35b4141e50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:06.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.728+0000 7f35bc5b1640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b413f0f0 con 0x7f35b413ece0 2024-03-28T16:15:06.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.728+0000 7f35bc5b1640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:06.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.728+0000 7f35bc5b1640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b4071d80 con 0x7f35b40719a0 2024-03-28T16:15:06.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.728+0000 7f35bc5b1640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:06.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.728+0000 7f35bc5b1640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b40aa3f0 con 0x7f35b4141e50 2024-03-28T16:15:06.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.728+0000 7f35bc5b1640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:06.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.728+0000 7f35bc5b1640 10 monclient(hunting): authenticate will time out at 1373.870390s 2024-03-28T16:15:06.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.729+0000 7f35ba34d640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f35b40719a0 0x7f35b413e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:06.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.729+0000 7f35bab4e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f35b4141e50 0x7f35b414be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:06.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.729+0000 7f35b9b4c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f35b413ece0 0x7f35b4141740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59804/0 (socket says 172.21.15.7:59804) 2024-03-28T16:15:06.734 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.729+0000 7f35b9b4c640 1 -- 172.21.15.7:0/1637162371 learned_addr learned my addr 172.21.15.7:0/1637162371 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-28T16:15:06.734 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.729+0000 7f35ba34d640 10 monclient(hunting): get_auth_request con 0x7f35b40719a0 auth_method 0 2024-03-28T16:15:06.734 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.729+0000 7f35ba34d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:06.734 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.729+0000 7f35ba34d640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:06.734 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.729+0000 7f35ba34d640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:06.734 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.729+0000 7f35bab4e640 10 monclient(hunting): get_auth_request con 0x7f35b4141e50 auth_method 0 2024-03-28T16:15:06.734 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.729+0000 7f35bab4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:06.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.729+0000 7f35bab4e640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:06.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.729+0000 7f35bab4e640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:06.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.730+0000 7f35ba34d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:06.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.730+0000 7f35ba34d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:06.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.730+0000 7f35ba34d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:06.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.730+0000 7f35bab4e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:06.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.730+0000 7f35bab4e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:06.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.730+0000 7f35bab4e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:06.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.732+0000 7f35ba34d640 10 monclient(hunting): handle_auth_done global_id 4221 payload 274 2024-03-28T16:15:06.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.732+0000 7f35ba34d640 1 -- 172.21.15.7:0/1637162371 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f35b4141e50 msgr2=0x7f35b414be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:06.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.732+0000 7f35ba34d640 1 --2- 172.21.15.7:0/1637162371 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f35b4141e50 0x7f35b414be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:06.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.732+0000 7f35ba34d640 1 -- 172.21.15.7:0/1637162371 >> v1:172.21.15.7:6789/0 conn(0x7f35b413ece0 legacy=0x7f35b4141740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:06.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.732+0000 7f35ba34d640 10 monclient: _finish_hunting 0 2024-03-28T16:15:06.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.732+0000 7f35ba34d640 1 monclient: found mon.noname-a 2024-03-28T16:15:06.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.732+0000 7f35ba34d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:06.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.732+0000 7f35ba34d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:15:06.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.732+0000 7f35ba34d640 1 -- 172.21.15.7:0/1637162371 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b40aa6d0 con 0x7f35b40719a0 2024-03-28T16:15:06.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.734+0000 7f35ba34d640 1 --2- 172.21.15.7:0/1637162371 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35b40719a0 0x7f35b413e7a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f35b00097a0 tx=0x7f35b002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82083de7ce11ac57 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:06.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.739+0000 7f35b934b640 1 -- 172.21.15.7:0/1637162371 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b002f050 con 0x7f35b40719a0 2024-03-28T16:15:06.747 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.739+0000 7f35b934b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:06.747 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.739+0000 7f35b934b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:15:06.747 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.739+0000 7f35b934b640 10 monclient: dump: 2024-03-28T16:15:06.747 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:06.747 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:06.747 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:06.747 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:06.747 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:06.747 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:06.747 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:06.747 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:06.747 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:06.747 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:06.747 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.739+0000 7f35b934b640 10 monclient: _finish_auth 0 2024-03-28T16:15:06.748 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.739+0000 7f35b934b640 10 monclient: _check_auth_tickets 2024-03-28T16:15:06.748 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.739+0000 7f35b934b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:06.748 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.739+0000 7f35b934b640 1 -- 172.21.15.7:0/1637162371 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b0031700 con 0x7f35b40719a0 2024-03-28T16:15:06.748 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.739+0000 7f35b934b640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:06.748 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.739+0000 7f35b934b640 1 -- 172.21.15.7:0/1637162371 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b0031a90 con 0x7f35b40719a0 2024-03-28T16:15:06.748 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.739+0000 7f35b934b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:06.748 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.739+0000 7f35b934b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:15:06.748 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.739+0000 7f35b934b640 10 monclient: dump: 2024-03-28T16:15:06.748 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:06.748 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:06.748 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:06.748 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:06.748 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:06.748 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:06.748 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:06.749 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:06.749 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:06.749 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:06.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.739+0000 7f35bc5b1640 5 monclient: authenticate success, global_id 4221 2024-03-28T16:15:06.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.739+0000 7f35bc5b1640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:06.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.739+0000 7f35bc5b1640 4 set_mon_vals no callback set 2024-03-28T16:15:06.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.740+0000 7f35bc5b1640 10 monclient: shutdown 2024-03-28T16:15:06.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.740+0000 7f35bc5b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:06.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.740+0000 7f35bc5b1640 1 -- 172.21.15.7:0/1637162371 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35b40719a0 msgr2=0x7f35b413e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:06.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.740+0000 7f35bc5b1640 1 --2- 172.21.15.7:0/1637162371 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35b40719a0 0x7f35b413e7a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f35b00097a0 tx=0x7f35b002b7a0 comp rx=0 tx=0).stop 2024-03-28T16:15:06.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.740+0000 7f35bc5b1640 1 -- 172.21.15.7:0/1637162371 shutdown_connections 2024-03-28T16:15:06.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.740+0000 7f35bc5b1640 1 --2- 172.21.15.7:0/1637162371 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f35b4141e50 0x7f35b414be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:06.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.740+0000 7f35bc5b1640 1 --2- 172.21.15.7:0/1637162371 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35b40719a0 0x7f35b413e7a0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f35b00097a0 tx=0x7f35b002b7a0 comp rx=0 tx=0).stop 2024-03-28T16:15:06.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.740+0000 7f35bc5b1640 1 -- 172.21.15.7:0/1637162371 >> 172.21.15.7:0/1637162371 conn(0x7f35b40b72b0 msgr2=0x7f35b40b76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:06.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.740+0000 7f35bc5b1640 1 -- 172.21.15.7:0/1637162371 shutdown_connections 2024-03-28T16:15:06.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.740+0000 7f35bc5b1640 1 -- 172.21.15.7:0/1637162371 wait complete. 2024-03-28T16:15:06.750 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.741+0000 7f35bc5b1640 10 monclient: build_initial_monmap 2024-03-28T16:15:06.750 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.741+0000 7f35bc5b1640 10 monclient: monmap: 2024-03-28T16:15:06.750 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-28T16:15:06.750 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:06.750 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:06.750 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-28T16:15:06.750 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:06.750 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:06.750 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:06.750 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:06.750 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:06.750 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:06.750 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.741+0000 7f35bc5b1640 1 librados: starting msgr at 2024-03-28T16:15:06.750 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.742+0000 7f35bc5b1640 1 librados: starting objecter 2024-03-28T16:15:06.750 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.742+0000 7f35bc5b1640 1 Processor -- start 2024-03-28T16:15:06.751 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.742+0000 7f35bc5b1640 1 -- start start 2024-03-28T16:15:06.751 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.742+0000 7f35bc5b1640 1 librados: setting wanted keys 2024-03-28T16:15:06.751 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.742+0000 7f35bc5b1640 1 librados: calling monclient init 2024-03-28T16:15:06.751 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.742+0000 7f35bc5b1640 10 monclient: init 2024-03-28T16:15:06.751 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.743+0000 7f35bc5b1640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:06.751 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.743+0000 7f35bc5b1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-28T16:15:06.751 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.743+0000 7f35bc5b1640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f35b413ece0 0x7f35b413d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:06.751 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.743+0000 7f35bc5b1640 10 monclient(hunting): picked mon.noname-c con 0x7f35b413ece0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:06.751 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.743+0000 7f35bc5b1640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35b4141e50 0x7f35b4143f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:06.751 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.743+0000 7f35bc5b1640 10 monclient(hunting): picked mon.noname-a con 0x7f35b4141e50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:06.751 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.743+0000 7f35bc5b1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35b4142880 0x7f35b4144440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:06.751 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.743+0000 7f35bc5b1640 10 monclient(hunting): picked mon.noname-b con 0x7f35b4142880 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:06.751 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.743+0000 7f35bc5b1640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:06.751 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.743+0000 7f35bc5b1640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b413cf90 con 0x7f35b4141e50 2024-03-28T16:15:06.752 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.743+0000 7f35bc5b1640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:06.752 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.743+0000 7f35bc5b1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b4143ae0 con 0x7f35b4142880 2024-03-28T16:15:06.752 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.743+0000 7f35bc5b1640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:06.752 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.743+0000 7f35bc5b1640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b413c3c0 con 0x7f35b413ece0 2024-03-28T16:15:06.752 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.743+0000 7f35bc5b1640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:06.752 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.743+0000 7f35bc5b1640 10 monclient(hunting): authenticate will time out at 1373.885146s 2024-03-28T16:15:06.752 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.743+0000 7f35ba34d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f35b413ece0 0x7f35b413d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:06.752 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.744+0000 7f35b9b4c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35b4141e50 0x7f35b4143f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:06.753 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.744+0000 7f35bab4e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35b4142880 0x7f35b4144440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:06.753 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.744+0000 7f35b9b4c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35b4141e50 0x7f35b4143f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50466/0 (socket says 172.21.15.7:50466) 2024-03-28T16:15:06.753 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.744+0000 7f35b9b4c640 1 -- 172.21.15.7:0/529494340 learned_addr learned my addr 172.21.15.7:0/529494340 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:06.753 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.744+0000 7f35b9b4c640 10 monclient(hunting): get_auth_request con 0x7f35b4141e50 auth_method 0 2024-03-28T16:15:06.753 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.744+0000 7f35b9b4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:06.753 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.744+0000 7f35b9b4c640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:06.753 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.744+0000 7f35b9b4c640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:06.753 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.744+0000 7f35bab4e640 10 monclient(hunting): get_auth_request con 0x7f35b4142880 auth_method 0 2024-03-28T16:15:06.753 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.744+0000 7f35bab4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:06.754 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.744+0000 7f35bab4e640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:06.754 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.744+0000 7f35bab4e640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:06.754 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.744+0000 7f35ba34d640 10 monclient(hunting): get_auth_request con 0x7f35b413ece0 auth_method 0 2024-03-28T16:15:06.754 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.744+0000 7f35ba34d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:06.754 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.744+0000 7f35ba34d640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:06.754 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.744+0000 7f35ba34d640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:06.754 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.745+0000 7f35bab4e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:06.754 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.745+0000 7f35bab4e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:06.754 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.745+0000 7f35bab4e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:06.755 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.745+0000 7f35b9b4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:06.755 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.745+0000 7f35b9b4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:06.755 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.745+0000 7f35b9b4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:06.755 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.745+0000 7f35ba34d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:06.755 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.745+0000 7f35ba34d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:06.755 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.745+0000 7f35ba34d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:06.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.747+0000 7f35bab4e640 10 monclient(hunting): handle_auth_done global_id 4216 payload 824 2024-03-28T16:15:06.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.748+0000 7f35bab4e640 1 -- 172.21.15.7:0/529494340 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f35b413ece0 msgr2=0x7f35b413d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:06.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.748+0000 7f35bab4e640 1 --2- 172.21.15.7:0/529494340 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f35b413ece0 0x7f35b413d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:06.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.748+0000 7f35bab4e640 1 -- 172.21.15.7:0/529494340 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35b4141e50 msgr2=0x7f35b4143f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:06.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.748+0000 7f35bab4e640 1 --2- 172.21.15.7:0/529494340 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35b4141e50 0x7f35b4143f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:06.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.748+0000 7f35bab4e640 10 monclient: _finish_hunting 0 2024-03-28T16:15:06.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.748+0000 7f35bab4e640 1 monclient: found mon.noname-b 2024-03-28T16:15:06.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.748+0000 7f35bab4e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:06.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.748+0000 7f35bab4e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:15:06.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.748+0000 7f35bab4e640 1 -- 172.21.15.7:0/529494340 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b0009400 con 0x7f35b4142880 2024-03-28T16:15:06.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.750+0000 7f35bab4e640 1 --2- 172.21.15.7:0/529494340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35b4142880 0x7f35b4144440 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f35ac00eba0 tx=0x7f35ac00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:06.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.750+0000 7f35aaffd640 10 client.?.objecter ms_handle_connect 0x7f35b4142880 2024-03-28T16:15:06.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.750+0000 7f35aaffd640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:06.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35aaffd640 1 -- 172.21.15.7:0/529494340 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ac00cd20 con 0x7f35b4142880 2024-03-28T16:15:06.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:06.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35aaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:06.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35aaffd640 10 monclient: dump: 2024-03-28T16:15:06.760 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:06.761 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:06.761 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:06.761 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:06.761 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:06.761 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:06.761 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:06.761 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:06.761 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:06.761 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:06.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35aaffd640 10 monclient: _finish_auth 0 2024-03-28T16:15:06.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35aaffd640 10 monclient: _check_auth_tickets 2024-03-28T16:15:06.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:06.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35aaffd640 1 -- 172.21.15.7:0/529494340 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35ac00f040 con 0x7f35b4142880 2024-03-28T16:15:06.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35bc5b1640 5 monclient: authenticate success, global_id 4216 2024-03-28T16:15:06.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35bc5b1640 10 monclient: _renew_subs 2024-03-28T16:15:06.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35bc5b1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:06.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35bc5b1640 1 -- 172.21.15.7:0/529494340 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b4144980 con 0x7f35b4142880 2024-03-28T16:15:06.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:06.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35aaffd640 1 -- 172.21.15.7:0/529494340 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ac0146c0 con 0x7f35b4142880 2024-03-28T16:15:06.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:06.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:06.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35aaffd640 10 monclient: dump: 2024-03-28T16:15:06.762 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:06.762 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:06.762 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:06.762 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:06.762 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:06.763 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:06.763 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:06.763 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:06.763 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:06.763 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:06.763 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35b8b4a640 4 set_mon_vals no callback set 2024-03-28T16:15:06.763 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35bc5b1640 10 client.4216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:06.763 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35bc5b1640 10 monclient: _renew_subs 2024-03-28T16:15:06.763 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35bc5b1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:06.763 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35bc5b1640 1 -- 172.21.15.7:0/529494340 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b413dec0 con 0x7f35b4142880 2024-03-28T16:15:06.763 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.754+0000 7f35bc5b1640 1 librados: init done 2024-03-28T16:15:06.763 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.755+0000 7f35bc5b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:06.764 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.755+0000 7f35bc5b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:06.764 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.755+0000 7f35bc5b1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:06.764 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.755+0000 7f35bc5b1640 1 -- 172.21.15.7:0/529494340 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b413d100 con 0x7f35b4142880 2024-03-28T16:15:06.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.778+0000 7f35aaffd640 1 -- 172.21.15.7:0/529494340 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 6) v1 ==== 89081+0+0 (secure 0 0 0) 0x7f35ac014820 con 0x7f35b4142880 2024-03-28T16:15:06.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.778+0000 7f35aaffd640 1 --2- 172.21.15.7:0/529494340 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f359806c530 0x7f359806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:06.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.778+0000 7f35ba34d640 1 --2- 172.21.15.7:0/529494340 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f359806c530 0x7f359806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:06.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.779+0000 7f35ba34d640 10 monclient: get_auth_request con 0x7f359806c530 auth_method 0 2024-03-28T16:15:06.781 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.779+0000 7f35aaffd640 10 client.4216.objecter ms_handle_connect 0x7f359806c530 2024-03-28T16:15:06.782 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:06.779+0000 7f35ba34d640 1 --2- 172.21.15.7:0/529494340 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f359806c530 0x7f359806e9f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f35b0035000 tx=0x7f35b0008030 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:06.871 INFO:tasks.ceph.osd.4.smithi083.stderr:==00:00:00:16.981 35294== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:06.872 INFO:tasks.ceph.osd.7.smithi083.stderr:==00:00:00:16.972 35295== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:06.968 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:17.082 34918== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:07.115 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.113+0000 7f35aaffd640 1 -- 172.21.15.7:0/529494340 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f35ac059390 con 0x7f35b4142880 2024-03-28T16:15:07.115 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.113+0000 7f35aaffd640 10 client.4216.objecter ms_dispatch 0x7f35b4000b90 osd_map(11..11 src has 1..11) v4 2024-03-28T16:15:07.115 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.113+0000 7f35aaffd640 3 client.4216.objecter handle_osd_map got epochs [11,11] > 0 2024-03-28T16:15:07.116 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.113+0000 7f35aaffd640 3 client.4216.objecter handle_osd_map decoding full epoch 11 2024-03-28T16:15:07.116 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.113+0000 7f35aaffd640 20 client.4216.objecter dump_active .. 0 homeless 2024-03-28T16:15:07.116 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.113+0000 7f35aaffd640 1 -- 172.21.15.7:0/529494340 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f35ac0556d0 con 0x7f35b4142880 2024-03-28T16:15:07.116 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.113+0000 7f35aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:07.116 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.113+0000 7f35aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:07.142 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:17.252 34919== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:07.172 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:17.265 34920== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:07.232 INFO:tasks.ceph.osd.6.smithi083.stderr:==00:00:00:17.317 35297== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:07.389 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:17.480 34921== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:07.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.450+0000 7f35bc5b1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:07.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.450+0000 7f35bc5b1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:07.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.450+0000 7f35bc5b1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:07.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.450+0000 7f35bc5b1640 1 -- 172.21.15.7:0/529494340 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35b4143c20 con 0x7f35b4142880 2024-03-28T16:15:07.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.466+0000 7f35aaffd640 1 -- 172.21.15.7:0/529494340 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f35b4143c20 con 0x7f35b4142880 2024-03-28T16:15:07.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.466+0000 7f35aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:07.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.466+0000 7f35aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:07.468 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-28T16:15:07.468 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":11,"fsid":"808ff1a1-2b17-4c35-a89a-340dbbf674ee","created":"2024-03-28T16:14:44.591599+0000","modified":"2024-03-28T16:14:56.773245+0000","last_up_change":"0.000000","last_in_change":"2024-03-28T16:14:49.757700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a19c6856-78b9-4919-97ce-6940a5af982c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2afee73d-3369-4fd2-bfc5-36dee610daf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0473d84c-e864-4373-b9d1-283130278363","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b64fd5e7-cdec-4bd0-b5e2-dfc8d4f6b9ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8f7552db-5651-430f-a826-3c6843bffb99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2aabdbd2-d870-4d6d-823e-89b008ea3108","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fbbb7557-08d4-48d4-9b88-6bb1d680a87d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"805f54d9-03d5-4b23-aef3-e1d55bb09f87","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:15:07.471 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.469+0000 7f35bc5b1640 10 librados: watch_flush enter 2024-03-28T16:15:07.471 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.469+0000 7f35bc5b1640 10 librados: watch_flush exit 2024-03-28T16:15:07.471 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.469+0000 7f35bc5b1640 20 client.4216.objecter shutdown clearing up homeless session... 2024-03-28T16:15:07.471 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.469+0000 7f35bc5b1640 10 client.4216.objecter successfully canceled tick 2024-03-28T16:15:07.471 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.469+0000 7f35bc5b1640 1 -- 172.21.15.7:0/529494340 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f359806c530 msgr2=0x7f359806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:07.471 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.469+0000 7f35bc5b1640 1 --2- 172.21.15.7:0/529494340 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f359806c530 0x7f359806e9f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f35b0035000 tx=0x7f35b0008030 comp rx=0 tx=0).stop 2024-03-28T16:15:07.471 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.469+0000 7f35bc5b1640 10 monclient: shutdown 2024-03-28T16:15:07.471 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.469+0000 7f35bc5b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:07.471 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.469+0000 7f35bc5b1640 1 -- 172.21.15.7:0/529494340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35b4142880 msgr2=0x7f35b4144440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:07.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.469+0000 7f35bc5b1640 1 --2- 172.21.15.7:0/529494340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35b4142880 0x7f35b4144440 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f35ac00eba0 tx=0x7f35ac00c490 comp rx=0 tx=0).stop 2024-03-28T16:15:07.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.469+0000 7f35bc5b1640 1 -- 172.21.15.7:0/529494340 shutdown_connections 2024-03-28T16:15:07.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.469+0000 7f35bc5b1640 1 --2- 172.21.15.7:0/529494340 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f359806c530 0x7f359806e9f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:07.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.469+0000 7f35bc5b1640 1 --2- 172.21.15.7:0/529494340 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35b4141e50 0x7f35b4143f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:07.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.469+0000 7f35bc5b1640 1 --2- 172.21.15.7:0/529494340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35b4142880 0x7f35b4144440 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:07.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.469+0000 7f35bc5b1640 1 --2- 172.21.15.7:0/529494340 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f35b413ece0 0x7f35b413d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:07.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.469+0000 7f35bc5b1640 1 -- 172.21.15.7:0/529494340 >> 172.21.15.7:0/529494340 conn(0x7f35b40b72b0 msgr2=0x7f35b40b4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:07.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.469+0000 7f35bc5b1640 1 -- 172.21.15.7:0/529494340 shutdown_connections 2024-03-28T16:15:07.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.469+0000 7f35bc5b1640 1 -- 172.21.15.7:0/529494340 wait complete. 2024-03-28T16:15:07.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:07.470+0000 7f35bc5b1640 1 librados: shutdown 2024-03-28T16:15:07.483 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-28T16:15:08.218 INFO:tasks.ceph.osd.5.smithi083.stderr:==00:00:00:18.301 35296== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:12.239 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:22.353 34918== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:12.245 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:22.359 34918== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:12.443 INFO:tasks.ceph.osd.7.smithi083.stderr:==00:00:00:22.543 35295== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:12.447 INFO:tasks.ceph.osd.7.smithi083.stderr:==00:00:00:22.547 35295== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:12.594 INFO:tasks.ceph.osd.4.smithi083.stderr:==00:00:00:22.704 35294== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:12.600 INFO:tasks.ceph.osd.4.smithi083.stderr:==00:00:00:22.710 35294== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:12.715 INFO:tasks.ceph.osd.6.smithi083.stderr:==00:00:00:22.800 35297== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:12.719 INFO:tasks.ceph.osd.6.smithi083.stderr:==00:00:00:22.804 35297== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:12.839 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:22.933 34920== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:12.843 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:22.938 34920== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:12.967 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:23.057 34921== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:12.973 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:23.064 34921== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:13.120 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:23.230 34919== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:13.126 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:23.235 34919== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:13.925 INFO:tasks.ceph.osd.5.smithi083.stderr:==00:00:00:24.008 35296== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:13.929 INFO:tasks.ceph.osd.5.smithi083.stderr:==00:00:00:24.012 35296== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:14.290 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-28T16:15:14.411 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.411+0000 7fbc8040b640 10 monclient: get_monmap_and_config 2024-03-28T16:15:14.412 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.411+0000 7fbc8040b640 10 monclient: build_initial_monmap 2024-03-28T16:15:14.412 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.411+0000 7fbc8040b640 10 monclient: monmap: 2024-03-28T16:15:14.412 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-28T16:15:14.412 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:14.412 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:15:14.412406+0000 2024-03-28T16:15:14.412 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:15:14.412406+0000 2024-03-28T16:15:14.412 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:14.412 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:14.412 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:14.412 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:14.412 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:14.412 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:14.412 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:14.412 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:14.415 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.412+0000 7fbc8040b640 1 Processor -- start 2024-03-28T16:15:14.415 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.412+0000 7fbc8040b640 1 -- start start 2024-03-28T16:15:14.415 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.412+0000 7fbc8040b640 10 monclient: init 2024-03-28T16:15:14.415 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.413+0000 7fbc8040b640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:14.415 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.413+0000 7fbc8040b640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-03-28T16:15:14.415 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.413+0000 7fbc8040b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fbc780719a0 0x7fbc7813e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:14.415 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.413+0000 7fbc8040b640 10 monclient(hunting): picked mon.noname-c con 0x7fbc780719a0 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:14.415 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.413+0000 7fbc8040b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc7813ece0 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:14.415 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.413+0000 7fbc8040b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc78141e50 addr v1:172.21.15.83:6789/0 2024-03-28T16:15:14.415 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.413+0000 7fbc8040b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc7813f0f0 con 0x7fbc7813ece0 2024-03-28T16:15:14.415 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.413+0000 7fbc8040b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc780aa3f0 con 0x7fbc78141e50 2024-03-28T16:15:14.415 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.413+0000 7fbc8040b640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:14.415 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.413+0000 7fbc8040b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc78071d80 con 0x7fbc780719a0 2024-03-28T16:15:14.416 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.413+0000 7fbc8040b640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:14.416 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.413+0000 7fbc8040b640 10 monclient(hunting): authenticate will time out at 1381.554770s 2024-03-28T16:15:14.416 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.414+0000 7fbc7d9a6640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbc7813ece0 0x7fbc78141740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59816/0 (socket says 172.21.15.7:59816) 2024-03-28T16:15:14.416 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.414+0000 7fbc7d9a6640 1 -- 172.21.15.7:0/2871573810 learned_addr learned my addr 172.21.15.7:0/2871573810 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-28T16:15:14.416 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.414+0000 7fbc7e1a7640 1 --2- 172.21.15.7:0/2871573810 >> v2:172.21.15.83:3300/0 conn(0x7fbc780719a0 0x7fbc7813e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:14.416 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.415+0000 7fbc7e1a7640 10 monclient(hunting): get_auth_request con 0x7fbc780719a0 auth_method 0 2024-03-28T16:15:14.416 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.415+0000 7fbc7e1a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:14.416 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.415+0000 7fbc7e1a7640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:14.416 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.415+0000 7fbc7e1a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:14.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.415+0000 7fbc7e1a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:14.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.415+0000 7fbc7e1a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:14.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.415+0000 7fbc7e1a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:14.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.417+0000 7fbc7d1a5640 1 -- 172.21.15.7:0/2871573810 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393284227 0 0) 0x7fbc7813f0f0 con 0x7fbc7813ece0 2024-03-28T16:15:14.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.417+0000 7fbc7d1a5640 10 monclient(hunting): _negotiate 2024-03-28T16:15:14.418 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.417+0000 7fbc7d1a5640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:14.418 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.417+0000 7fbc7d1a5640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:14.418 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.417+0000 7fbc7d1a5640 10 monclient(hunting): my global_id is 4227 2024-03-28T16:15:14.418 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.417+0000 7fbc7d1a5640 1 -- 172.21.15.7:0/2871573810 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc64003520 con 0x7fbc7813ece0 2024-03-28T16:15:14.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 1 -- 172.21.15.7:0/2871573810 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3476514225 0 0) 0x7fbc64003520 con 0x7fbc7813ece0 2024-03-28T16:15:14.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 1 -- 172.21.15.7:0/2871573810 >> v2:172.21.15.83:3300/0 conn(0x7fbc780719a0 msgr2=0x7fbc7813e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:14.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 1 --2- 172.21.15.7:0/2871573810 >> v2:172.21.15.83:3300/0 conn(0x7fbc780719a0 0x7fbc7813e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:14.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 1 -- 172.21.15.7:0/2871573810 >> v1:172.21.15.83:6789/0 conn(0x7fbc78141e50 legacy=0x7fbc7814be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:14.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 10 monclient: _finish_hunting 0 2024-03-28T16:15:14.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 1 monclient: found mon.noname-a-legacy 2024-03-28T16:15:14.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:14.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.7:6789/0 2024-03-28T16:15:14.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 1 -- 172.21.15.7:0/2871573810 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc780aa6d0 con 0x7fbc7813ece0 2024-03-28T16:15:14.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 10 monclient: _finish_auth 0 2024-03-28T16:15:14.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 10 monclient: _check_auth_tickets 2024-03-28T16:15:14.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:14.422 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 1 -- 172.21.15.7:0/2871573810 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1105724391 0 0) 0x7fbc68001690 con 0x7fbc7813ece0 2024-03-28T16:15:14.422 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:14.422 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-03-28T16:15:14.422 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 10 monclient: dump: 2024-03-28T16:15:14.422 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:14.422 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:14.422 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:14.422 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:14.422 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:14.422 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:14.422 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:14.422 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:14.422 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:14.422 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:14.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] but i'm connected to v1:172.21.15.7:6789/0, reconnecting 2024-03-28T16:15:14.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:14.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 1 -- 172.21.15.7:0/2871573810 >> v1:172.21.15.7:6789/0 conn(0x7fbc7813ece0 legacy=0x7fbc78141740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:14.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-28T16:15:14.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 1 --2- 172.21.15.7:0/2871573810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc640039e0 0x7fbc64003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:14.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 10 monclient(hunting): picked mon.a con 0x7fbc640039e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:14.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 1 --2- 172.21.15.7:0/2871573810 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbc640047c0 0x7fbc64004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:14.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 10 monclient(hunting): picked mon.c con 0x7fbc640047c0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:14.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 1 --2- 172.21.15.7:0/2871573810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc640051f0 0x7fbc6400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:14.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 10 monclient(hunting): picked mon.b con 0x7fbc640051f0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:14.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:14.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 1 -- 172.21.15.7:0/2871573810 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc78071d80 con 0x7fbc640039e0 2024-03-28T16:15:14.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:14.424 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 1 -- 172.21.15.7:0/2871573810 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6400c150 con 0x7fbc640051f0 2024-03-28T16:15:14.424 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:14.424 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 1 -- 172.21.15.7:0/2871573810 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6400c310 con 0x7fbc640047c0 2024-03-28T16:15:14.424 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:14.424 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7d1a5640 10 monclient(hunting): _finish_auth 0 2024-03-28T16:15:14.424 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.420+0000 7fbc7e1a7640 1 --2- 172.21.15.7:0/2871573810 >> v2:172.21.15.83:3300/0 conn(0x7fbc780719a0 0x7fbc7813e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:15:14.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.422+0000 7fbc7d9a6640 1 --2- 172.21.15.7:0/2871573810 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbc640047c0 0x7fbc64004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:14.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.422+0000 7fbc7d9a6640 10 monclient(hunting): get_auth_request con 0x7fbc640047c0 auth_method 0 2024-03-28T16:15:14.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.422+0000 7fbc7d9a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:14.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.422+0000 7fbc7d9a6640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:14.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.422+0000 7fbc7d9a6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:15:14.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.422+0000 7fbc7e1a7640 1 --2- 172.21.15.7:0/2871573810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc640039e0 0x7fbc64003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:14.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.423+0000 7fbc7e1a7640 10 monclient(hunting): get_auth_request con 0x7fbc640039e0 auth_method 0 2024-03-28T16:15:14.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.423+0000 7fbc7e1a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:14.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.423+0000 7fbc7e1a7640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:14.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.423+0000 7fbc7e1a7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:15:14.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.423+0000 7fbc7e9a8640 1 --2- 172.21.15.7:0/2871573810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc640051f0 0x7fbc6400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:14.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.423+0000 7fbc7e1a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:14.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.423+0000 7fbc7e1a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:14.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.423+0000 7fbc7e1a7640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-28T16:15:14.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.423+0000 7fbc7d9a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:14.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.423+0000 7fbc7d9a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:14.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.423+0000 7fbc7d9a6640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-28T16:15:14.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.425+0000 7fbc7e1a7640 10 monclient(hunting): handle_auth_done global_id 4227 payload 293 2024-03-28T16:15:14.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.425+0000 7fbc7e1a7640 1 -- 172.21.15.7:0/2871573810 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbc640047c0 msgr2=0x7fbc64004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:14.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.425+0000 7fbc7e1a7640 1 --2- 172.21.15.7:0/2871573810 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbc640047c0 0x7fbc64004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:14.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.425+0000 7fbc7e1a7640 1 -- 172.21.15.7:0/2871573810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc640051f0 msgr2=0x7fbc6400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:14.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.425+0000 7fbc7e1a7640 1 --2- 172.21.15.7:0/2871573810 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc640051f0 0x7fbc6400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:14.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.425+0000 7fbc7e1a7640 10 monclient: _finish_hunting 0 2024-03-28T16:15:14.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.425+0000 7fbc7e9a8640 1 -- 172.21.15.7:0/2871573810 reap_dead start 2024-03-28T16:15:14.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.425+0000 7fbc7e1a7640 1 monclient: found mon.a 2024-03-28T16:15:14.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.425+0000 7fbc7e1a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:14.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.425+0000 7fbc7e1a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:14.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.425+0000 7fbc7e1a7640 1 -- 172.21.15.7:0/2871573810 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc64003520 con 0x7fbc640039e0 2024-03-28T16:15:14.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.425+0000 7fbc7e1a7640 10 monclient: _finish_auth 0 2024-03-28T16:15:14.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.425+0000 7fbc7e1a7640 10 monclient: _check_auth_tickets 2024-03-28T16:15:14.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.425+0000 7fbc7e1a7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:14.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.425+0000 7fbc8040b640 5 monclient: authenticate success, global_id 4227 2024-03-28T16:15:14.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.425+0000 7fbc8040b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:14.429 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.427+0000 7fbc7e1a7640 1 --2- 172.21.15.7:0/2871573810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc640039e0 0x7fbc64003e00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbc74009c70 tx=0x7fbc640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc4b61e680e60e76 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:14.433 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.431+0000 7fbc7d1a5640 1 -- 172.21.15.7:0/2871573810 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc74004020 con 0x7fbc640039e0 2024-03-28T16:15:14.433 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.431+0000 7fbc7d1a5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:14.433 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.431+0000 7fbc7d1a5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:15:14.433 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.431+0000 7fbc7d1a5640 10 monclient: dump: 2024-03-28T16:15:14.433 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:14.433 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:14.433 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:14.434 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:14.434 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:14.434 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:14.434 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:14.434 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:14.434 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:14.434 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:14.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.431+0000 7fbc7d1a5640 1 -- 172.21.15.7:0/2871573810 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc74004180 con 0x7fbc640039e0 2024-03-28T16:15:14.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.431+0000 7fbc7d1a5640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:14.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.431+0000 7fbc7d1a5640 1 -- 172.21.15.7:0/2871573810 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7402c860 con 0x7fbc640039e0 2024-03-28T16:15:14.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.431+0000 7fbc7d1a5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:14.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.431+0000 7fbc7d1a5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:15:14.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.431+0000 7fbc7d1a5640 10 monclient: dump: 2024-03-28T16:15:14.434 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:14.435 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:14.435 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:14.435 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:14.435 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:14.435 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:14.435 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:14.435 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:14.435 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:14.435 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:14.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.431+0000 7fbc8040b640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:14.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.431+0000 7fbc8040b640 4 set_mon_vals no callback set 2024-03-28T16:15:14.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.432+0000 7fbc8040b640 10 monclient: shutdown 2024-03-28T16:15:14.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.432+0000 7fbc8040b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:14.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.432+0000 7fbc8040b640 1 -- 172.21.15.7:0/2871573810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc640039e0 msgr2=0x7fbc64003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:14.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.432+0000 7fbc8040b640 1 --2- 172.21.15.7:0/2871573810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc640039e0 0x7fbc64003e00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbc74009c70 tx=0x7fbc640057f0 comp rx=0 tx=0).stop 2024-03-28T16:15:14.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.432+0000 7fbc8040b640 1 -- 172.21.15.7:0/2871573810 shutdown_connections 2024-03-28T16:15:14.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.432+0000 7fbc8040b640 1 --2- 172.21.15.7:0/2871573810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc640039e0 0x7fbc64003e00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:14.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.433+0000 7fbc8040b640 1 -- 172.21.15.7:0/2871573810 >> 172.21.15.7:0/2871573810 conn(0x7fbc780b72b0 msgr2=0x7fbc780b76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:14.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.433+0000 7fbc8040b640 1 -- 172.21.15.7:0/2871573810 shutdown_connections 2024-03-28T16:15:14.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.433+0000 7fbc8040b640 1 -- 172.21.15.7:0/2871573810 wait complete. 2024-03-28T16:15:14.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.434+0000 7fbc8040b640 10 monclient: build_initial_monmap 2024-03-28T16:15:14.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.434+0000 7fbc8040b640 10 monclient: monmap: 2024-03-28T16:15:14.442 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-28T16:15:14.442 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:14.442 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:14.442 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-28T16:15:14.442 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:14.442 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:14.442 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:14.442 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:14.442 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:14.442 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:14.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.434+0000 7fbc8040b640 1 librados: starting msgr at 2024-03-28T16:15:14.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.434+0000 7fbc8040b640 1 librados: starting objecter 2024-03-28T16:15:14.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.435+0000 7fbc8040b640 1 Processor -- start 2024-03-28T16:15:14.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.435+0000 7fbc8040b640 1 -- start start 2024-03-28T16:15:14.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.435+0000 7fbc8040b640 1 librados: setting wanted keys 2024-03-28T16:15:14.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.435+0000 7fbc8040b640 1 librados: calling monclient init 2024-03-28T16:15:14.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.435+0000 7fbc8040b640 10 monclient: init 2024-03-28T16:15:14.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.436+0000 7fbc8040b640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:14.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.436+0000 7fbc8040b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-28T16:15:14.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.436+0000 7fbc8040b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc640039e0 0x7fbc781452e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:14.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.436+0000 7fbc8040b640 10 monclient(hunting): picked mon.noname-b con 0x7fbc640039e0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:14.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.436+0000 7fbc8040b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc781428b0 0x7fbc780ab180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:14.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.436+0000 7fbc8040b640 10 monclient(hunting): picked mon.noname-a con 0x7fbc781428b0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:14.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.436+0000 7fbc8040b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbc781d6de0 0x7fbc781d7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:14.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc7e1a7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc640039e0 0x7fbc781452e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:14.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc7d9a6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc781428b0 0x7fbc780ab180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:14.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc7d9a6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc781428b0 0x7fbc780ab180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50482/0 (socket says 172.21.15.7:50482) 2024-03-28T16:15:14.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc8040b640 10 monclient(hunting): picked mon.noname-c con 0x7fbc781d6de0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:14.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc8040b640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:14.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc8040b640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc780ab6c0 con 0x7fbc781428b0 2024-03-28T16:15:14.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc8040b640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:14.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc8040b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc780ab900 con 0x7fbc640039e0 2024-03-28T16:15:14.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc8040b640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:14.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc8040b640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc781d6950 con 0x7fbc781d6de0 2024-03-28T16:15:14.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc8040b640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:14.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc8040b640 10 monclient(hunting): authenticate will time out at 1381.578372s 2024-03-28T16:15:14.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc7d9a6640 1 -- 172.21.15.7:0/3484575416 learned_addr learned my addr 172.21.15.7:0/3484575416 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:14.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc7d9a6640 10 monclient(hunting): get_auth_request con 0x7fbc781428b0 auth_method 0 2024-03-28T16:15:14.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc7d9a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:14.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc7d9a6640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:14.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc7d9a6640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:14.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc7e1a7640 10 monclient(hunting): get_auth_request con 0x7fbc640039e0 auth_method 0 2024-03-28T16:15:14.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc7e1a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:14.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc7e1a7640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:14.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc7e1a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:14.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.437+0000 7fbc7e9a8640 1 --2- 172.21.15.7:0/3484575416 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbc781d6de0 0x7fbc781d7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:14.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.438+0000 7fbc7d9a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:14.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.438+0000 7fbc7d9a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:14.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.438+0000 7fbc7d9a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:14.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.438+0000 7fbc7e9a8640 10 monclient(hunting): get_auth_request con 0x7fbc781d6de0 auth_method 0 2024-03-28T16:15:14.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.438+0000 7fbc7e9a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:14.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.438+0000 7fbc7e9a8640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:14.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.438+0000 7fbc7e9a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:14.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.438+0000 7fbc7e1a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:14.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.438+0000 7fbc7e1a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:14.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.438+0000 7fbc7e1a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:14.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.439+0000 7fbc7e9a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:14.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.439+0000 7fbc7e9a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:14.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.439+0000 7fbc7e9a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:14.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.440+0000 7fbc7d9a6640 10 monclient(hunting): handle_auth_done global_id 4230 payload 824 2024-03-28T16:15:14.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.440+0000 7fbc7d9a6640 1 -- 172.21.15.7:0/3484575416 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbc781d6de0 msgr2=0x7fbc781d7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:14.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.440+0000 7fbc7d9a6640 1 --2- 172.21.15.7:0/3484575416 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbc781d6de0 0x7fbc781d7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:14.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.440+0000 7fbc7d9a6640 1 -- 172.21.15.7:0/3484575416 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc640039e0 msgr2=0x7fbc781452e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:14.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.440+0000 7fbc7d9a6640 1 --2- 172.21.15.7:0/3484575416 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc640039e0 0x7fbc781452e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:14.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.440+0000 7fbc7d9a6640 10 monclient: _finish_hunting 0 2024-03-28T16:15:14.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.440+0000 7fbc7d9a6640 1 monclient: found mon.noname-a 2024-03-28T16:15:14.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.440+0000 7fbc7d9a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:14.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.440+0000 7fbc7d9a6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:15:14.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.440+0000 7fbc7d9a6640 1 -- 172.21.15.7:0/3484575416 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc780ac860 con 0x7fbc781428b0 2024-03-28T16:15:14.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.442+0000 7fbc7d9a6640 1 --2- 172.21.15.7:0/3484575416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc781428b0 0x7fbc780ab180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbc68007290 tx=0x7fbc68007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:14.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.442+0000 7fbc6effd640 10 client.?.objecter ms_handle_connect 0x7fbc781428b0 2024-03-28T16:15:14.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.443+0000 7fbc6effd640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:14.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc6effd640 1 -- 172.21.15.7:0/3484575416 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6800f960 con 0x7fbc781428b0 2024-03-28T16:15:14.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:14.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc6effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:15:14.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc6effd640 10 monclient: dump: 2024-03-28T16:15:14.448 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:14.448 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:14.448 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:14.448 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:14.449 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:14.449 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:14.449 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:14.449 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:14.449 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:14.449 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:14.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc6effd640 10 monclient: _finish_auth 0 2024-03-28T16:15:14.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc6effd640 10 monclient: _check_auth_tickets 2024-03-28T16:15:14.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:14.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc6effd640 1 -- 172.21.15.7:0/3484575416 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc6800fac0 con 0x7fbc781428b0 2024-03-28T16:15:14.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc8040b640 5 monclient: authenticate success, global_id 4230 2024-03-28T16:15:14.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc8040b640 10 monclient: _renew_subs 2024-03-28T16:15:14.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc8040b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:14.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc8040b640 1 -- 172.21.15.7:0/3484575416 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc781444c0 con 0x7fbc781428b0 2024-03-28T16:15:14.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:14.450 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc6effd640 1 -- 172.21.15.7:0/3484575416 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6800fe70 con 0x7fbc781428b0 2024-03-28T16:15:14.450 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:14.450 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc6effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:15:14.450 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc6effd640 10 monclient: dump: 2024-03-28T16:15:14.450 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:14.450 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:14.450 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:14.450 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:14.450 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:14.450 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:14.450 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:14.450 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:14.450 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:14.450 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:14.450 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc8040b640 10 client.4230.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:14.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc8040b640 10 monclient: _renew_subs 2024-03-28T16:15:14.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc7c9a4640 4 set_mon_vals no callback set 2024-03-28T16:15:14.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc8040b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:14.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc8040b640 1 -- 172.21.15.7:0/3484575416 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc780b0250 con 0x7fbc781428b0 2024-03-28T16:15:14.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.446+0000 7fbc8040b640 1 librados: init done 2024-03-28T16:15:14.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.448+0000 7fbc8040b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:14.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.448+0000 7fbc8040b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:14.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.448+0000 7fbc8040b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:14.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.448+0000 7fbc8040b640 1 -- 172.21.15.7:0/3484575416 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc78069430 con 0x7fbc781428b0 2024-03-28T16:15:14.485 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.484+0000 7fbc6effd640 1 -- 172.21.15.7:0/3484575416 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 6) v1 ==== 89081+0+0 (secure 0 0 0) 0x7fbc6801d460 con 0x7fbc781428b0 2024-03-28T16:15:14.485 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.485+0000 7fbc6effd640 1 --2- 172.21.15.7:0/3484575416 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fbc5c06c530 0x7fbc5c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:14.485 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.485+0000 7fbc7e1a7640 1 --2- 172.21.15.7:0/3484575416 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fbc5c06c530 0x7fbc5c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:14.485 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.485+0000 7fbc7e1a7640 10 monclient: get_auth_request con 0x7fbc5c06c530 auth_method 0 2024-03-28T16:15:14.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.485+0000 7fbc7e1a7640 1 --2- 172.21.15.7:0/3484575416 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fbc5c06c530 0x7fbc5c06e9f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbc74035000 tx=0x7fbc74030000 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:14.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.486+0000 7fbc6effd640 10 client.4230.objecter ms_handle_connect 0x7fbc5c06c530 2024-03-28T16:15:14.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.932+0000 7fbc6effd640 1 -- 172.21.15.7:0/3484575416 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fbc68058e20 con 0x7fbc781428b0 2024-03-28T16:15:14.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.932+0000 7fbc6effd640 10 client.4230.objecter ms_dispatch 0x7fbc78000b90 osd_map(11..11 src has 1..11) v4 2024-03-28T16:15:14.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.932+0000 7fbc6effd640 3 client.4230.objecter handle_osd_map got epochs [11,11] > 0 2024-03-28T16:15:14.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.932+0000 7fbc6effd640 3 client.4230.objecter handle_osd_map decoding full epoch 11 2024-03-28T16:15:14.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.933+0000 7fbc6effd640 20 client.4230.objecter dump_active .. 0 homeless 2024-03-28T16:15:14.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.933+0000 7fbc6effd640 1 -- 172.21.15.7:0/3484575416 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbc68055160 con 0x7fbc781428b0 2024-03-28T16:15:14.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.933+0000 7fbc6effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:14.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:14.933+0000 7fbc6effd640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:15.180 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.180+0000 7fbc8040b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:15.181 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.180+0000 7fbc8040b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:15.181 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.180+0000 7fbc8040b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:15.181 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.180+0000 7fbc8040b640 1 -- 172.21.15.7:0/3484575416 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc781d6a90 con 0x7fbc781428b0 2024-03-28T16:15:15.208 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.208+0000 7fbc6effd640 1 -- 172.21.15.7:0/3484575416 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fbc68058780 con 0x7fbc781428b0 2024-03-28T16:15:15.208 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.208+0000 7fbc6effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:15.208 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.208+0000 7fbc6effd640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:15.208 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-28T16:15:15.208 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":11,"fsid":"808ff1a1-2b17-4c35-a89a-340dbbf674ee","created":"2024-03-28T16:14:44.591599+0000","modified":"2024-03-28T16:14:56.773245+0000","last_up_change":"0.000000","last_in_change":"2024-03-28T16:14:49.757700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a19c6856-78b9-4919-97ce-6940a5af982c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2afee73d-3369-4fd2-bfc5-36dee610daf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0473d84c-e864-4373-b9d1-283130278363","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b64fd5e7-cdec-4bd0-b5e2-dfc8d4f6b9ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8f7552db-5651-430f-a826-3c6843bffb99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2aabdbd2-d870-4d6d-823e-89b008ea3108","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fbbb7557-08d4-48d4-9b88-6bb1d680a87d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"805f54d9-03d5-4b23-aef3-e1d55bb09f87","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:15:15.212 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.211+0000 7fbc8040b640 10 librados: watch_flush enter 2024-03-28T16:15:15.212 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.211+0000 7fbc8040b640 10 librados: watch_flush exit 2024-03-28T16:15:15.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.211+0000 7fbc8040b640 20 client.4230.objecter shutdown clearing up homeless session... 2024-03-28T16:15:15.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.211+0000 7fbc8040b640 10 client.4230.objecter successfully canceled tick 2024-03-28T16:15:15.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.211+0000 7fbc8040b640 1 -- 172.21.15.7:0/3484575416 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fbc5c06c530 msgr2=0x7fbc5c06e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:15.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.211+0000 7fbc8040b640 1 --2- 172.21.15.7:0/3484575416 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fbc5c06c530 0x7fbc5c06e9f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbc74035000 tx=0x7fbc74030000 comp rx=0 tx=0).stop 2024-03-28T16:15:15.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.211+0000 7fbc8040b640 10 monclient: shutdown 2024-03-28T16:15:15.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.211+0000 7fbc8040b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:15.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.211+0000 7fbc8040b640 1 -- 172.21.15.7:0/3484575416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc781428b0 msgr2=0x7fbc780ab180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:15.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.211+0000 7fbc8040b640 1 --2- 172.21.15.7:0/3484575416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc781428b0 0x7fbc780ab180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbc68007290 tx=0x7fbc68007750 comp rx=0 tx=0).stop 2024-03-28T16:15:15.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.211+0000 7fbc8040b640 1 -- 172.21.15.7:0/3484575416 shutdown_connections 2024-03-28T16:15:15.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.211+0000 7fbc8040b640 1 --2- 172.21.15.7:0/3484575416 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fbc5c06c530 0x7fbc5c06e9f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:15.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.211+0000 7fbc8040b640 1 --2- 172.21.15.7:0/3484575416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc781428b0 0x7fbc780ab180 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:15.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.211+0000 7fbc8040b640 1 --2- 172.21.15.7:0/3484575416 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbc781d6de0 0x7fbc781d7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:15.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.211+0000 7fbc8040b640 1 --2- 172.21.15.7:0/3484575416 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc640039e0 0x7fbc781452e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:15.214 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.211+0000 7fbc8040b640 1 -- 172.21.15.7:0/3484575416 >> 172.21.15.7:0/3484575416 conn(0x7fbc780b72b0 msgr2=0x7fbc78068ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:15.214 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.211+0000 7fbc8040b640 1 -- 172.21.15.7:0/3484575416 shutdown_connections 2024-03-28T16:15:15.214 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.211+0000 7fbc8040b640 1 -- 172.21.15.7:0/3484575416 wait complete. 2024-03-28T16:15:15.214 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:15.212+0000 7fbc8040b640 1 librados: shutdown 2024-03-28T16:15:15.225 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-28T16:15:16.568 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:26.682 34918== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:16.601 INFO:tasks.ceph.osd.7.smithi083.stderr:==00:00:00:26.700 35295== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:16.808 INFO:tasks.ceph.osd.4.smithi083.stderr:==00:00:00:26.918 35294== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:17.104 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:27.198 34920== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:17.181 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:27.271 34921== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:17.209 INFO:tasks.ceph.osd.6.smithi083.stderr:==00:00:00:27.294 35297== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:17.497 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:27.607 34919== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:18.150 INFO:tasks.ceph.osd.5.smithi083.stderr:==00:00:00:28.233 35296== Warning: unimplemented fcntl command: 1038 2024-03-28T16:15:22.032 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-28T16:15:22.164 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.163+0000 7f0695ba2640 10 monclient: get_monmap_and_config 2024-03-28T16:15:22.164 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.163+0000 7f0695ba2640 10 monclient: build_initial_monmap 2024-03-28T16:15:22.164 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.163+0000 7f0695ba2640 10 monclient: monmap: 2024-03-28T16:15:22.164 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-28T16:15:22.164 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:22.164 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:15:22.164459+0000 2024-03-28T16:15:22.164 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:15:22.164459+0000 2024-03-28T16:15:22.164 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:22.164 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:22.164 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:22.164 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:22.165 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:22.165 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:22.165 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:22.165 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:22.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.164+0000 7f0695ba2640 1 Processor -- start 2024-03-28T16:15:22.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.165+0000 7f0695ba2640 1 -- start start 2024-03-28T16:15:22.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.165+0000 7f0695ba2640 10 monclient: init 2024-03-28T16:15:22.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.165+0000 7f0695ba2640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:22.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.165+0000 7f0695ba2640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-03-28T16:15:22.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.165+0000 7f0695ba2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f06900b3d50 0x7f069009bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:22.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.165+0000 7f0695ba2640 10 monclient(hunting): picked mon.noname-c con 0x7f06900b3d50 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:22.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.165+0000 7f0695ba2640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f06900b1080 0x7f06900b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:22.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.165+0000 7f0695ba2640 10 monclient(hunting): picked mon.noname-a con 0x7f06900b1080 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:22.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.165+0000 7f0695ba2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f069013e580 addr v1:172.21.15.83:6789/0 2024-03-28T16:15:22.169 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.165+0000 7f0695ba2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069009b790 con 0x7f069013e580 2024-03-28T16:15:22.169 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.165+0000 7f0695ba2640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:22.169 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.165+0000 7f0695ba2640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0690099e60 con 0x7f06900b1080 2024-03-28T16:15:22.169 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.165+0000 7f0695ba2640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:22.169 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.165+0000 7f0695ba2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f069009c040 con 0x7f06900b3d50 2024-03-28T16:15:22.169 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.165+0000 7f0695ba2640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:22.169 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.165+0000 7f0695ba2640 10 monclient(hunting): authenticate will time out at 1389.306828s 2024-03-28T16:15:22.169 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.166+0000 7f068ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f06900b1080 0x7f06900b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:22.169 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.167+0000 7f0694ba0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f06900b3d50 0x7f069009bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:22.169 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.167+0000 7f068ffff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f06900b1080 0x7f06900b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55434/0 (socket says 172.21.15.7:55434) 2024-03-28T16:15:22.169 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.167+0000 7f068ffff640 1 -- 172.21.15.7:0/4190516858 learned_addr learned my addr 172.21.15.7:0/4190516858 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:22.169 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.167+0000 7f068ffff640 10 monclient(hunting): get_auth_request con 0x7f06900b1080 auth_method 0 2024-03-28T16:15:22.170 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.167+0000 7f068ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:22.170 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.167+0000 7f068ffff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:22.170 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.167+0000 7f068ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:22.170 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.167+0000 7f0694ba0640 10 monclient(hunting): get_auth_request con 0x7f06900b3d50 auth_method 0 2024-03-28T16:15:22.170 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.167+0000 7f0694ba0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:22.170 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.167+0000 7f0694ba0640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:22.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.167+0000 7f0694ba0640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:22.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.167+0000 7f068ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:22.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.167+0000 7f068ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:22.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.167+0000 7f068ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:22.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.168+0000 7f0694ba0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:22.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.169+0000 7f0694ba0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:22.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.169+0000 7f0694ba0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:22.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.170+0000 7f068ffff640 10 monclient(hunting): handle_auth_done global_id 4233 payload 274 2024-03-28T16:15:22.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.170+0000 7f068ffff640 1 -- 172.21.15.7:0/4190516858 >> v2:172.21.15.83:3300/0 conn(0x7f06900b3d50 msgr2=0x7f069009bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:22.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.170+0000 7f068ffff640 1 --2- 172.21.15.7:0/4190516858 >> v2:172.21.15.83:3300/0 conn(0x7f06900b3d50 0x7f069009bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:22.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.170+0000 7f068ffff640 1 -- 172.21.15.7:0/4190516858 >> v1:172.21.15.83:6789/0 conn(0x7f069013e580 legacy=0x7f069013e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:22.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.170+0000 7f068ffff640 10 monclient: _finish_hunting 0 2024-03-28T16:15:22.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.170+0000 7f068ffff640 1 monclient: found mon.noname-a 2024-03-28T16:15:22.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.170+0000 7f068ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:22.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.170+0000 7f068ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:15:22.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.170+0000 7f068ffff640 1 -- 172.21.15.7:0/4190516858 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0690099a60 con 0x7f06900b1080 2024-03-28T16:15:22.174 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.173+0000 7f068ffff640 1 --2- 172.21.15.7:0/4190516858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06900b1080 0x7f06900b2e90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f069006b2e0 tx=0x7f067c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9188bc0909fb8ca server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:22.176 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.176+0000 7f068f7fe640 1 -- 172.21.15.7:0/4190516858 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c030860 con 0x7f06900b1080 2024-03-28T16:15:22.176 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.176+0000 7f068f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:22.176 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.176+0000 7f068f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:15:22.176 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.176+0000 7f068f7fe640 10 monclient: dump: 2024-03-28T16:15:22.176 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:22.176 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:22.176 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:22.176 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:22.176 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:22.177 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:22.177 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:22.177 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:22.177 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:22.177 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:22.177 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.176+0000 7f068f7fe640 10 monclient: _finish_auth 0 2024-03-28T16:15:22.177 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.176+0000 7f068f7fe640 10 monclient: _check_auth_tickets 2024-03-28T16:15:22.177 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.176+0000 7f068f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:22.177 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.176+0000 7f0695ba2640 5 monclient: authenticate success, global_id 4233 2024-03-28T16:15:22.177 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.176+0000 7f0695ba2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:22.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.179+0000 7f068f7fe640 1 -- 172.21.15.7:0/4190516858 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067c0309c0 con 0x7f06900b1080 2024-03-28T16:15:22.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.179+0000 7f068f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:22.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.179+0000 7f068f7fe640 1 -- 172.21.15.7:0/4190516858 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067c030d50 con 0x7f06900b1080 2024-03-28T16:15:22.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.179+0000 7f0695ba2640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:22.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.179+0000 7f0695ba2640 4 set_mon_vals no callback set 2024-03-28T16:15:22.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.179+0000 7f068f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:22.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.179+0000 7f068f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:15:22.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.179+0000 7f068f7fe640 10 monclient: dump: 2024-03-28T16:15:22.190 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:22.190 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:22.191 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:22.191 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:22.191 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:22.191 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:22.191 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:22.191 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:22.191 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:22.191 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:22.191 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.180+0000 7f0695ba2640 10 monclient: shutdown 2024-03-28T16:15:22.191 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.180+0000 7f0695ba2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:22.191 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.180+0000 7f0695ba2640 1 -- 172.21.15.7:0/4190516858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06900b1080 msgr2=0x7f06900b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:22.191 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.180+0000 7f0695ba2640 1 --2- 172.21.15.7:0/4190516858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06900b1080 0x7f06900b2e90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f069006b2e0 tx=0x7f067c02b9a0 comp rx=0 tx=0).stop 2024-03-28T16:15:22.191 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.180+0000 7f0695ba2640 1 -- 172.21.15.7:0/4190516858 shutdown_connections 2024-03-28T16:15:22.191 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.180+0000 7f0695ba2640 1 --2- 172.21.15.7:0/4190516858 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06900b1080 0x7f06900b2e90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:22.191 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.180+0000 7f0695ba2640 1 --2- 172.21.15.7:0/4190516858 >> v2:172.21.15.83:3300/0 conn(0x7f06900b3d50 0x7f069009bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:22.192 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.180+0000 7f0695ba2640 1 -- 172.21.15.7:0/4190516858 >> 172.21.15.7:0/4190516858 conn(0x7f069006a570 msgr2=0x7f069006a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:22.192 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.180+0000 7f0695ba2640 1 -- 172.21.15.7:0/4190516858 shutdown_connections 2024-03-28T16:15:22.192 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.180+0000 7f0695ba2640 1 -- 172.21.15.7:0/4190516858 wait complete. 2024-03-28T16:15:22.192 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.181+0000 7f0695ba2640 10 monclient: build_initial_monmap 2024-03-28T16:15:22.192 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.181+0000 7f0695ba2640 10 monclient: monmap: 2024-03-28T16:15:22.192 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-28T16:15:22.192 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:22.192 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:22.192 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-28T16:15:22.192 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:22.192 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:22.192 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:22.192 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:22.192 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:22.193 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:22.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.181+0000 7f0695ba2640 1 librados: starting msgr at 2024-03-28T16:15:22.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.181+0000 7f0695ba2640 1 librados: starting objecter 2024-03-28T16:15:22.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.182+0000 7f0695ba2640 1 Processor -- start 2024-03-28T16:15:22.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.182+0000 7f0695ba2640 1 -- start start 2024-03-28T16:15:22.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.182+0000 7f0695ba2640 1 librados: setting wanted keys 2024-03-28T16:15:22.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.182+0000 7f0695ba2640 1 librados: calling monclient init 2024-03-28T16:15:22.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.182+0000 7f0695ba2640 10 monclient: init 2024-03-28T16:15:22.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.183+0000 7f0695ba2640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:22.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.183+0000 7f0695ba2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-28T16:15:22.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.183+0000 7f0695ba2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06900b1080 0x7f0690141620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:22.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.183+0000 7f0695ba2640 10 monclient(hunting): picked mon.noname-b con 0x7f06900b1080 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:22.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.183+0000 7f0695ba2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06900b3d50 0x7f0690144330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:22.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.183+0000 7f0695ba2640 10 monclient(hunting): picked mon.noname-a con 0x7f06900b3d50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:22.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.183+0000 7f0695ba2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f069013e580 0x7f06901366d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:22.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.183+0000 7f0695ba2640 10 monclient(hunting): picked mon.noname-c con 0x7f069013e580 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:22.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.183+0000 7f0695ba2640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:22.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.183+0000 7f0695ba2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0690144b50 con 0x7f06900b3d50 2024-03-28T16:15:22.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.183+0000 7f0695ba2640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:22.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.183+0000 7f0695ba2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0690144870 con 0x7f06900b1080 2024-03-28T16:15:22.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.183+0000 7f0695ba2640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:22.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.183+0000 7f0695ba2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06901455c0 con 0x7f069013e580 2024-03-28T16:15:22.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.183+0000 7f0695ba2640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:22.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.183+0000 7f0695ba2640 10 monclient(hunting): authenticate will time out at 1389.324519s 2024-03-28T16:15:22.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.183+0000 7f068ffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06900b3d50 0x7f0690144330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:22.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.183+0000 7f0694ba0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06900b1080 0x7f0690141620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:22.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.183+0000 7f06953a1640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f069013e580 0x7f06901366d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:22.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.184+0000 7f0694ba0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06900b1080 0x7f0690141620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:35076/0 (socket says 172.21.15.7:35076) 2024-03-28T16:15:22.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.184+0000 7f0694ba0640 1 -- 172.21.15.7:0/3356781408 learned_addr learned my addr 172.21.15.7:0/3356781408 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:22.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.184+0000 7f0694ba0640 10 monclient(hunting): get_auth_request con 0x7f06900b1080 auth_method 0 2024-03-28T16:15:22.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.184+0000 7f0694ba0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:22.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.184+0000 7f0694ba0640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:22.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.184+0000 7f0694ba0640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:22.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.184+0000 7f068ffff640 10 monclient(hunting): get_auth_request con 0x7f06900b3d50 auth_method 0 2024-03-28T16:15:22.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.184+0000 7f068ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:22.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.184+0000 7f068ffff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:22.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.184+0000 7f068ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:22.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.184+0000 7f06953a1640 10 monclient(hunting): get_auth_request con 0x7f069013e580 auth_method 0 2024-03-28T16:15:22.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.184+0000 7f06953a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:22.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.184+0000 7f06953a1640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:22.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.184+0000 7f06953a1640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:22.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.184+0000 7f0694ba0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:22.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.184+0000 7f0694ba0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:22.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.184+0000 7f0694ba0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:22.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.184+0000 7f068ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:22.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.184+0000 7f068ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:22.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.185+0000 7f068ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:22.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.185+0000 7f06953a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:22.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.185+0000 7f06953a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:22.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.185+0000 7f06953a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:22.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.187+0000 7f0694ba0640 10 monclient(hunting): handle_auth_done global_id 4234 payload 824 2024-03-28T16:15:22.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.187+0000 7f0694ba0640 1 -- 172.21.15.7:0/3356781408 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f069013e580 msgr2=0x7f06901366d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:22.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.187+0000 7f0694ba0640 1 --2- 172.21.15.7:0/3356781408 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f069013e580 0x7f06901366d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:22.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.187+0000 7f0694ba0640 1 -- 172.21.15.7:0/3356781408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06900b3d50 msgr2=0x7f0690144330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:22.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.187+0000 7f0694ba0640 1 --2- 172.21.15.7:0/3356781408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06900b3d50 0x7f0690144330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:22.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.187+0000 7f0694ba0640 10 monclient: _finish_hunting 0 2024-03-28T16:15:22.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.187+0000 7f0694ba0640 1 monclient: found mon.noname-b 2024-03-28T16:15:22.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.187+0000 7f0694ba0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:22.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.187+0000 7f0694ba0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:15:22.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.187+0000 7f0694ba0640 1 -- 172.21.15.7:0/3356781408 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067c0094d0 con 0x7f06900b1080 2024-03-28T16:15:22.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.190+0000 7f0694ba0640 1 --2- 172.21.15.7:0/3356781408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06900b1080 0x7f0690141620 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f068400e9b0 tx=0x7f068400ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:22.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.190+0000 7f068d7fa640 10 client.?.objecter ms_handle_connect 0x7f06900b1080 2024-03-28T16:15:22.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.190+0000 7f068d7fa640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:22.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.192+0000 7f068d7fa640 1 -- 172.21.15.7:0/3356781408 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684009670 con 0x7f06900b1080 2024-03-28T16:15:22.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.192+0000 7f068d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:22.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.192+0000 7f068d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:22.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.192+0000 7f068d7fa640 10 monclient: dump: 2024-03-28T16:15:22.200 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:22.200 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:22.200 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:22.200 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:22.200 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:22.200 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:22.200 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:22.201 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:22.201 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:22.201 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:22.201 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.192+0000 7f068d7fa640 10 monclient: _finish_auth 0 2024-03-28T16:15:22.201 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.192+0000 7f068d7fa640 10 monclient: _check_auth_tickets 2024-03-28T16:15:22.201 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.192+0000 7f068d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:22.201 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.192+0000 7f0695ba2640 5 monclient: authenticate success, global_id 4234 2024-03-28T16:15:22.201 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.192+0000 7f0695ba2640 10 monclient: _renew_subs 2024-03-28T16:15:22.201 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.192+0000 7f0695ba2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:22.201 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.192+0000 7f0695ba2640 1 -- 172.21.15.7:0/3356781408 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06900b3550 con 0x7f06900b1080 2024-03-28T16:15:22.201 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.192+0000 7f0695ba2640 10 client.4234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:22.201 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.192+0000 7f0695ba2640 10 monclient: _renew_subs 2024-03-28T16:15:22.201 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.192+0000 7f0695ba2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:22.201 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.192+0000 7f0695ba2640 1 -- 172.21.15.7:0/3356781408 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0690136df0 con 0x7f06900b1080 2024-03-28T16:15:22.201 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.192+0000 7f0695ba2640 1 librados: init done 2024-03-28T16:15:22.202 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.194+0000 7f0695ba2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:22.202 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.194+0000 7f0695ba2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:22.202 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.194+0000 7f0695ba2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:22.202 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.194+0000 7f068d7fa640 1 -- 172.21.15.7:0/3356781408 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f068400c9a0 con 0x7f06900b1080 2024-03-28T16:15:22.202 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.194+0000 7f0695ba2640 1 -- 172.21.15.7:0/3356781408 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0690074a00 con 0x7f06900b1080 2024-03-28T16:15:22.202 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.194+0000 7f068d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:22.202 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.194+0000 7f068effd640 4 set_mon_vals no callback set 2024-03-28T16:15:22.202 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.195+0000 7f068d7fa640 1 -- 172.21.15.7:0/3356781408 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068400cd10 con 0x7f06900b1080 2024-03-28T16:15:22.202 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.195+0000 7f068d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:22.202 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.195+0000 7f068d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:22.202 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.195+0000 7f068d7fa640 10 monclient: dump: 2024-03-28T16:15:22.202 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:22.202 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:22.202 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:22.202 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:22.203 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:22.203 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:22.203 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:22.203 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:22.203 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:22.203 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:22.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.463+0000 7f068d7fa640 1 -- 172.21.15.7:0/3356781408 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 6) v1 ==== 89081+0+0 (secure 0 0 0) 0x7f0684013470 con 0x7f06900b1080 2024-03-28T16:15:22.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.464+0000 7f068d7fa640 1 --2- 172.21.15.7:0/3356781408 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f067406c500 0x7f067406e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:22.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.464+0000 7f068ffff640 1 --2- 172.21.15.7:0/3356781408 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f067406c500 0x7f067406e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:22.465 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.464+0000 7f068ffff640 10 monclient: get_auth_request con 0x7f067406c500 auth_method 0 2024-03-28T16:15:22.465 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.465+0000 7f068ffff640 1 --2- 172.21.15.7:0/3356781408 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f067406c500 0x7f067406e9c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f067c036000 tx=0x7f067c008030 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:22.465 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.465+0000 7f068d7fa640 10 client.4234.objecter ms_handle_connect 0x7f067406c500 2024-03-28T16:15:22.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.476+0000 7f068d7fa640 1 -- 172.21.15.7:0/3356781408 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f06840587f0 con 0x7f06900b1080 2024-03-28T16:15:22.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.476+0000 7f068d7fa640 10 client.4234.objecter ms_dispatch 0x7f0690000b90 osd_map(11..11 src has 1..11) v4 2024-03-28T16:15:22.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.476+0000 7f068d7fa640 3 client.4234.objecter handle_osd_map got epochs [11,11] > 0 2024-03-28T16:15:22.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.476+0000 7f068d7fa640 3 client.4234.objecter handle_osd_map decoding full epoch 11 2024-03-28T16:15:22.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.476+0000 7f068d7fa640 20 client.4234.objecter dump_active .. 0 homeless 2024-03-28T16:15:22.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.477+0000 7f068d7fa640 1 -- 172.21.15.7:0/3356781408 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0684054b30 con 0x7f06900b1080 2024-03-28T16:15:22.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.477+0000 7f068d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:22.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.477+0000 7f068d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:22.713 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.713+0000 7f0695ba2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:22.714 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.713+0000 7f0695ba2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:22.714 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.713+0000 7f0695ba2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:22.714 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.713+0000 7f0695ba2640 1 -- 172.21.15.7:0/3356781408 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0690069430 con 0x7f06900b1080 2024-03-28T16:15:22.739 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.739+0000 7f068d7fa640 1 -- 172.21.15.7:0/3356781408 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f0684017090 con 0x7f06900b1080 2024-03-28T16:15:22.739 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.739+0000 7f068d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:22.739 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.739+0000 7f068d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:22.740 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-28T16:15:22.740 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":11,"fsid":"808ff1a1-2b17-4c35-a89a-340dbbf674ee","created":"2024-03-28T16:14:44.591599+0000","modified":"2024-03-28T16:14:56.773245+0000","last_up_change":"0.000000","last_in_change":"2024-03-28T16:14:49.757700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a19c6856-78b9-4919-97ce-6940a5af982c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2afee73d-3369-4fd2-bfc5-36dee610daf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0473d84c-e864-4373-b9d1-283130278363","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b64fd5e7-cdec-4bd0-b5e2-dfc8d4f6b9ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8f7552db-5651-430f-a826-3c6843bffb99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2aabdbd2-d870-4d6d-823e-89b008ea3108","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fbbb7557-08d4-48d4-9b88-6bb1d680a87d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"805f54d9-03d5-4b23-aef3-e1d55bb09f87","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:15:22.741 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.741+0000 7f0695ba2640 10 librados: watch_flush enter 2024-03-28T16:15:22.741 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.741+0000 7f0695ba2640 10 librados: watch_flush exit 2024-03-28T16:15:22.741 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.741+0000 7f0695ba2640 20 client.4234.objecter shutdown clearing up homeless session... 2024-03-28T16:15:22.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.741+0000 7f0695ba2640 10 client.4234.objecter successfully canceled tick 2024-03-28T16:15:22.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.741+0000 7f0695ba2640 1 -- 172.21.15.7:0/3356781408 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f067406c500 msgr2=0x7f067406e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:22.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.741+0000 7f0695ba2640 1 --2- 172.21.15.7:0/3356781408 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f067406c500 0x7f067406e9c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f067c036000 tx=0x7f067c008030 comp rx=0 tx=0).stop 2024-03-28T16:15:22.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.741+0000 7f0695ba2640 10 monclient: shutdown 2024-03-28T16:15:22.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.741+0000 7f0695ba2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:22.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.741+0000 7f0695ba2640 1 -- 172.21.15.7:0/3356781408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06900b1080 msgr2=0x7f0690141620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:22.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.741+0000 7f0695ba2640 1 --2- 172.21.15.7:0/3356781408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06900b1080 0x7f0690141620 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f068400e9b0 tx=0x7f068400ee70 comp rx=0 tx=0).stop 2024-03-28T16:15:22.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.741+0000 7f0695ba2640 1 -- 172.21.15.7:0/3356781408 shutdown_connections 2024-03-28T16:15:22.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.741+0000 7f0695ba2640 1 --2- 172.21.15.7:0/3356781408 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f067406c500 0x7f067406e9c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:22.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.741+0000 7f0695ba2640 1 --2- 172.21.15.7:0/3356781408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06900b3d50 0x7f0690144330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:22.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.741+0000 7f0695ba2640 1 --2- 172.21.15.7:0/3356781408 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f069013e580 0x7f06901366d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:22.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.741+0000 7f0695ba2640 1 --2- 172.21.15.7:0/3356781408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06900b1080 0x7f0690141620 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:22.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.741+0000 7f0695ba2640 1 -- 172.21.15.7:0/3356781408 >> 172.21.15.7:0/3356781408 conn(0x7f069006a570 msgr2=0x7f06900b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:22.743 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.742+0000 7f0695ba2640 1 -- 172.21.15.7:0/3356781408 shutdown_connections 2024-03-28T16:15:22.743 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.742+0000 7f0695ba2640 1 -- 172.21.15.7:0/3356781408 wait complete. 2024-03-28T16:15:22.743 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:22.742+0000 7f0695ba2640 1 librados: shutdown 2024-03-28T16:15:22.752 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-28T16:15:24.506 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:15:24.503+0000 5dc2480 -1 osd.7 0 log_to_monitors true 2024-03-28T16:15:24.802 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:15:24.794+0000 5dc2480 -1 osd.4 0 log_to_monitors true 2024-03-28T16:15:25.102 INFO:tasks.ceph.osd.6.smithi083.stderr:2024-03-28T16:15:25.099+0000 5dc2480 -1 osd.6 0 log_to_monitors true 2024-03-28T16:15:25.148 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-03-28T16:15:25.145+0000 5dc2480 -1 osd.0 0 log_to_monitors true 2024-03-28T16:15:26.019 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:15:26.016+0000 5dc2480 -1 osd.2 0 log_to_monitors true 2024-03-28T16:15:26.170 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:15:26.160+0000 5dc2480 -1 osd.3 0 log_to_monitors true 2024-03-28T16:15:26.360 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:15:26.311+0000 5dc2480 -1 osd.5 0 log_to_monitors true 2024-03-28T16:15:26.628 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-03-28T16:15:26.625+0000 5dc2480 -1 osd.1 0 log_to_monitors true 2024-03-28T16:15:27.934 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:15:27.931+0000 c497640 -1 osd.7 0 waiting for initial osdmap 2024-03-28T16:15:27.939 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:15:27.935+0000 ac7e640 -1 osd.4 0 waiting for initial osdmap 2024-03-28T16:15:28.355 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:15:28.341+0000 1025c640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T16:15:28.356 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:15:28.344+0000 1025c640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T16:15:29.013 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-03-28T16:15:29.009+0000 c497640 -1 osd.0 0 waiting for initial osdmap 2024-03-28T16:15:29.430 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-03-28T16:15:29.416+0000 1025c640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T16:15:29.559 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-28T16:15:29.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.655+0000 7f4568ba2640 10 monclient: get_monmap_and_config 2024-03-28T16:15:29.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.655+0000 7f4568ba2640 10 monclient: build_initial_monmap 2024-03-28T16:15:29.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.655+0000 7f4568ba2640 10 monclient: monmap: 2024-03-28T16:15:29.656 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-28T16:15:29.656 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:29.656 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:15:29.655866+0000 2024-03-28T16:15:29.656 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:15:29.655866+0000 2024-03-28T16:15:29.656 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:29.656 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:29.656 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:29.656 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:29.656 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:29.656 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:29.656 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:29.657 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:29.658 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.655+0000 7f4568ba2640 1 Processor -- start 2024-03-28T16:15:29.658 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.656+0000 7f4568ba2640 1 -- start start 2024-03-28T16:15:29.658 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.656+0000 7f4568ba2640 10 monclient: init 2024-03-28T16:15:29.658 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.656+0000 7f4568ba2640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:29.658 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.656+0000 7f4568ba2640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-03-28T16:15:29.658 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.656+0000 7f4568ba2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f456406a600 0x7f45640b3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:29.659 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.656+0000 7f4568ba2640 10 monclient(hunting): picked mon.noname-c con 0x7f456406a600 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:29.659 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.656+0000 7f4568ba2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4564099f80 addr v1:172.21.15.83:6789/0 2024-03-28T16:15:29.659 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.656+0000 7f4568ba2640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f456413e550 0x7f456413e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:29.659 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.656+0000 7f4568ba2640 10 monclient(hunting): picked mon.noname-a con 0x7f456413e550 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:29.659 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.656+0000 7f4568ba2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4564099af0 con 0x7f4564099f80 2024-03-28T16:15:29.659 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.656+0000 7f4568ba2640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:29.659 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.656+0000 7f4568ba2640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f456406a9e0 con 0x7f456413e550 2024-03-28T16:15:29.659 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.656+0000 7f4568ba2640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:29.659 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.656+0000 7f4568ba2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f456409a3d0 con 0x7f456406a600 2024-03-28T16:15:29.659 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.656+0000 7f4568ba2640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:29.659 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.656+0000 7f4568ba2640 10 monclient(hunting): authenticate will time out at 1396.797509s 2024-03-28T16:15:29.695 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.694+0000 7f4563fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f456413e550 0x7f456413e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:29.695 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.694+0000 7f4563fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f456413e550 0x7f456413e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51822/0 (socket says 172.21.15.7:51822) 2024-03-28T16:15:29.695 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.694+0000 7f4563fff640 1 -- 172.21.15.7:0/3757400340 learned_addr learned my addr 172.21.15.7:0/3757400340 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:29.695 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.694+0000 7f4563fff640 10 monclient(hunting): get_auth_request con 0x7f456413e550 auth_method 0 2024-03-28T16:15:29.695 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.694+0000 7f4563fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:29.695 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.694+0000 7f4563fff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:29.695 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.694+0000 7f4563fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:29.700 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.700+0000 7f45637fe640 1 --2- 172.21.15.7:0/3757400340 >> v2:172.21.15.83:3300/0 conn(0x7f456406a600 0x7f45640b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:29.701 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.701+0000 7f45637fe640 10 monclient(hunting): get_auth_request con 0x7f456406a600 auth_method 0 2024-03-28T16:15:29.701 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.701+0000 7f45637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:29.701 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.701+0000 7f45637fe640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:29.701 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.701+0000 7f45637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:29.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.702+0000 7f45637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:29.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.702+0000 7f45637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:29.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.702+0000 7f45637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:29.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.705+0000 7f45637fe640 10 monclient(hunting): handle_auth_done global_id 4261 payload 274 2024-03-28T16:15:29.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.705+0000 7f45637fe640 1 -- 172.21.15.7:0/3757400340 >> v2:172.21.15.7:3300/0 conn(0x7f456413e550 msgr2=0x7f456413e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:29.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.705+0000 7f45637fe640 1 --2- 172.21.15.7:0/3757400340 >> v2:172.21.15.7:3300/0 conn(0x7f456413e550 0x7f456413e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:29.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.705+0000 7f45637fe640 1 -- 172.21.15.7:0/3757400340 >> v1:172.21.15.83:6789/0 conn(0x7f4564099f80 legacy=0x7f456409bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:29.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.705+0000 7f45637fe640 10 monclient: _finish_hunting 0 2024-03-28T16:15:29.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.705+0000 7f45637fe640 1 monclient: found mon.noname-c 2024-03-28T16:15:29.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.705+0000 7f45637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:29.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.705+0000 7f45637fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3300/0 2024-03-28T16:15:29.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.705+0000 7f45637fe640 1 -- 172.21.15.7:0/3757400340 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f456409b470 con 0x7f456406a600 2024-03-28T16:15:29.711 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.711+0000 7f45637fe640 1 --2- 172.21.15.7:0/3757400340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f456406a600 0x7f45640b3170 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f45540097a0 tx=0x7f455402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce3ae2bc3f0f225f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:29.717 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.716+0000 7f45627fc640 1 -- 172.21.15.7:0/3757400340 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455402e070 con 0x7f456406a600 2024-03-28T16:15:29.717 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.716+0000 7f45627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:29.717 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.716+0000 7f45627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:29.717 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.716+0000 7f45627fc640 10 monclient: dump: 2024-03-28T16:15:29.717 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:29.717 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:29.718 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:29.718 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:29.718 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:29.718 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:29.718 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:29.718 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:29.718 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:29.718 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:29.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.716+0000 7f45627fc640 10 monclient: _finish_auth 0 2024-03-28T16:15:29.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.716+0000 7f45627fc640 10 monclient: _check_auth_tickets 2024-03-28T16:15:29.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.716+0000 7f45627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:29.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.716+0000 7f45627fc640 1 -- 172.21.15.7:0/3757400340 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45540315a0 con 0x7f456406a600 2024-03-28T16:15:29.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.716+0000 7f45627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:29.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.716+0000 7f45627fc640 1 -- 172.21.15.7:0/3757400340 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4554031930 con 0x7f456406a600 2024-03-28T16:15:29.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.716+0000 7f4568ba2640 5 monclient: authenticate success, global_id 4261 2024-03-28T16:15:29.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.716+0000 7f4568ba2640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:29.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.716+0000 7f4568ba2640 4 set_mon_vals no callback set 2024-03-28T16:15:29.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.716+0000 7f45627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:29.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.716+0000 7f45627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:29.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.716+0000 7f45627fc640 10 monclient: dump: 2024-03-28T16:15:29.719 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:29.719 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:29.719 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:29.719 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:29.719 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:29.719 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:29.719 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:29.719 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:29.720 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:29.720 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:29.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.717+0000 7f4568ba2640 10 monclient: shutdown 2024-03-28T16:15:29.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.717+0000 7f4568ba2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:29.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.717+0000 7f4568ba2640 1 -- 172.21.15.7:0/3757400340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f456406a600 msgr2=0x7f45640b3170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:29.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.717+0000 7f4568ba2640 1 --2- 172.21.15.7:0/3757400340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f456406a600 0x7f45640b3170 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f45540097a0 tx=0x7f455402b840 comp rx=0 tx=0).stop 2024-03-28T16:15:29.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.717+0000 7f4568ba2640 1 -- 172.21.15.7:0/3757400340 shutdown_connections 2024-03-28T16:15:29.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.717+0000 7f4568ba2640 1 --2- 172.21.15.7:0/3757400340 >> v2:172.21.15.7:3300/0 conn(0x7f456413e550 0x7f456413e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:29.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.717+0000 7f4568ba2640 1 --2- 172.21.15.7:0/3757400340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f456406a600 0x7f45640b3170 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:29.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.717+0000 7f4568ba2640 1 -- 172.21.15.7:0/3757400340 >> 172.21.15.7:0/3757400340 conn(0x7f45640b6f50 msgr2=0x7f45640b7330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:29.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.717+0000 7f4568ba2640 1 -- 172.21.15.7:0/3757400340 shutdown_connections 2024-03-28T16:15:29.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.717+0000 7f4568ba2640 1 -- 172.21.15.7:0/3757400340 wait complete. 2024-03-28T16:15:29.722 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.718+0000 7f4568ba2640 10 monclient: build_initial_monmap 2024-03-28T16:15:29.722 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.718+0000 7f4568ba2640 10 monclient: monmap: 2024-03-28T16:15:29.723 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-28T16:15:29.723 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:29.723 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:29.723 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-28T16:15:29.723 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:29.723 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:29.723 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:29.723 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:29.723 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:29.723 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:29.723 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.718+0000 7f4568ba2640 1 librados: starting msgr at 2024-03-28T16:15:29.723 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.718+0000 7f4568ba2640 1 librados: starting objecter 2024-03-28T16:15:29.723 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.718+0000 7f4568ba2640 1 Processor -- start 2024-03-28T16:15:29.723 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.719+0000 7f4568ba2640 1 -- start start 2024-03-28T16:15:29.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.719+0000 7f4568ba2640 1 librados: setting wanted keys 2024-03-28T16:15:29.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.719+0000 7f4568ba2640 1 librados: calling monclient init 2024-03-28T16:15:29.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.719+0000 7f4568ba2640 10 monclient: init 2024-03-28T16:15:29.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.719+0000 7f4568ba2640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:29.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.720+0000 7f4568ba2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-28T16:15:29.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.720+0000 7f4568ba2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f456406a600 0x7f4564133380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:29.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.720+0000 7f4568ba2640 10 monclient(hunting): picked mon.noname-b con 0x7f456406a600 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:29.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.720+0000 7f4568ba2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4564099f80 0x7f4564136e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:29.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.720+0000 7f4568ba2640 10 monclient(hunting): picked mon.noname-c con 0x7f4564099f80 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:29.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.720+0000 7f4568ba2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f456413e550 0x7f45641d96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:29.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.720+0000 7f4568ba2640 10 monclient(hunting): picked mon.noname-a con 0x7f456413e550 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:29.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.720+0000 7f4568ba2640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:29.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.720+0000 7f4568ba2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4564137590 con 0x7f456413e550 2024-03-28T16:15:29.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.720+0000 7f4568ba2640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:29.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.720+0000 7f4568ba2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4564137a20 con 0x7f456406a600 2024-03-28T16:15:29.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.720+0000 7f4568ba2640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:29.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.720+0000 7f4568ba2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4564133df0 con 0x7f4564099f80 2024-03-28T16:15:29.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.720+0000 7f4568ba2640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:29.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.720+0000 7f4568ba2640 10 monclient(hunting): authenticate will time out at 1396.861051s 2024-03-28T16:15:29.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.722+0000 7f4562ffd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4564099f80 0x7f4564136e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:29.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.722+0000 7f4562ffd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4564099f80 0x7f4564136e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:57890/0 (socket says 172.21.15.7:57890) 2024-03-28T16:15:29.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.722+0000 7f4562ffd640 1 -- 172.21.15.7:0/402325825 learned_addr learned my addr 172.21.15.7:0/402325825 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:29.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.722+0000 7f4562ffd640 10 monclient(hunting): get_auth_request con 0x7f4564099f80 auth_method 0 2024-03-28T16:15:29.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.722+0000 7f4562ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:29.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.722+0000 7f4562ffd640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:29.726 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.722+0000 7f4562ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:29.726 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.723+0000 7f4562ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:29.726 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.723+0000 7f4562ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:29.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.723+0000 7f4562ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:29.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.726+0000 7f45637fe640 1 --2- 172.21.15.7:0/402325825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f456406a600 0x7f4564133380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:29.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.726+0000 7f4562ffd640 10 monclient(hunting): handle_auth_done global_id 4229 payload 824 2024-03-28T16:15:29.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.726+0000 7f4562ffd640 1 -- 172.21.15.7:0/402325825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f456406a600 msgr2=0x7f4564133380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:29.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.726+0000 7f4562ffd640 1 --2- 172.21.15.7:0/402325825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f456406a600 0x7f4564133380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:29.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.726+0000 7f4562ffd640 1 -- 172.21.15.7:0/402325825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f456413e550 msgr2=0x7f45641d96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:29.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.726+0000 7f4562ffd640 1 --2- 172.21.15.7:0/402325825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f456413e550 0x7f45641d96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:29.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.726+0000 7f4562ffd640 10 monclient: _finish_hunting 0 2024-03-28T16:15:29.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.726+0000 7f4562ffd640 1 monclient: found mon.noname-c 2024-03-28T16:15:29.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.726+0000 7f4562ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:29.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.726+0000 7f4562ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-03-28T16:15:29.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.726+0000 7f4562ffd640 1 -- 172.21.15.7:0/402325825 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4554009400 con 0x7f4564099f80 2024-03-28T16:15:29.730 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.729+0000 7f4562ffd640 1 --2- 172.21.15.7:0/402325825 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4564099f80 0x7f4564136e10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f455800c400 tx=0x7f455800c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:29.730 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.729+0000 7f454bfff640 10 client.?.objecter ms_handle_connect 0x7f4564099f80 2024-03-28T16:15:29.730 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.729+0000 7f454bfff640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:29.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f454bfff640 1 -- 172.21.15.7:0/402325825 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45580052d0 con 0x7f4564099f80 2024-03-28T16:15:29.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f454bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:29.734 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f454bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:29.734 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f454bfff640 10 monclient: dump: 2024-03-28T16:15:29.734 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:29.734 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:29.734 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:29.734 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:29.734 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:29.734 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:29.734 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:29.734 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:29.735 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:29.735 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:29.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f454bfff640 10 monclient: _finish_auth 0 2024-03-28T16:15:29.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f454bfff640 10 monclient: _check_auth_tickets 2024-03-28T16:15:29.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f454bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:29.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f454bfff640 1 -- 172.21.15.7:0/402325825 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4558005450 con 0x7f4564099f80 2024-03-28T16:15:29.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f4568ba2640 5 monclient: authenticate success, global_id 4229 2024-03-28T16:15:29.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f454bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:29.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f4568ba2640 10 monclient: _renew_subs 2024-03-28T16:15:29.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f454bfff640 1 -- 172.21.15.7:0/402325825 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4558007260 con 0x7f4564099f80 2024-03-28T16:15:29.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f4568ba2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:29.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f4568ba2640 1 -- 172.21.15.7:0/402325825 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4564136500 con 0x7f4564099f80 2024-03-28T16:15:29.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f454bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:29.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f454bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:15:29.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f454bfff640 10 monclient: dump: 2024-03-28T16:15:29.736 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:29.736 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:29.736 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:29.736 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:29.736 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:29.736 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:29.736 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:29.736 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:29.736 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:29.736 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:29.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f4561ffb640 4 set_mon_vals no callback set 2024-03-28T16:15:29.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f4568ba2640 10 client.4229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:29.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f4568ba2640 10 monclient: _renew_subs 2024-03-28T16:15:29.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f4568ba2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:29.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f4568ba2640 1 -- 172.21.15.7:0/402325825 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45641d9e10 con 0x7f4564099f80 2024-03-28T16:15:29.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.733+0000 7f4568ba2640 1 librados: init done 2024-03-28T16:15:29.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.734+0000 7f4568ba2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:29.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.734+0000 7f4568ba2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:29.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.734+0000 7f4568ba2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:29.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.734+0000 7f4568ba2640 1 -- 172.21.15.7:0/402325825 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4564137350 con 0x7f4564099f80 2024-03-28T16:15:29.752 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.752+0000 7f454bfff640 1 -- 172.21.15.7:0/402325825 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 6) v1 ==== 89081+0+0 (secure 0 0 0) 0x7f45580073c0 con 0x7f4564099f80 2024-03-28T16:15:29.753 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.752+0000 7f454bfff640 1 --2- 172.21.15.7:0/402325825 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f455006c550 0x7f455006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:29.753 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.753+0000 7f45637fe640 1 --2- 172.21.15.7:0/402325825 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f455006c550 0x7f455006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:29.753 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.753+0000 7f45637fe640 10 monclient: get_auth_request con 0x7f455006c550 auth_method 0 2024-03-28T16:15:29.753 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.753+0000 7f45637fe640 1 --2- 172.21.15.7:0/402325825 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f455006c550 0x7f455006ea10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4554036000 tx=0x7f455403d040 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:29.753 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:29.753+0000 7f454bfff640 10 client.4229.objecter ms_handle_connect 0x7f455006c550 2024-03-28T16:15:30.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.038+0000 7f454bfff640 1 -- 172.21.15.7:0/402325825 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2447+0+0 (secure 0 0 0) 0x7f4558007670 con 0x7f4564099f80 2024-03-28T16:15:30.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.038+0000 7f454bfff640 10 client.4229.objecter ms_dispatch 0x7f4564000b90 osd_map(14..14 src has 1..14) v4 2024-03-28T16:15:30.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.038+0000 7f454bfff640 3 client.4229.objecter handle_osd_map got epochs [14,14] > 0 2024-03-28T16:15:30.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.038+0000 7f454bfff640 3 client.4229.objecter handle_osd_map decoding full epoch 14 2024-03-28T16:15:30.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.038+0000 7f454bfff640 20 client.4229.objecter dump_active .. 0 homeless 2024-03-28T16:15:30.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.038+0000 7f454bfff640 1 -- 172.21.15.7:0/402325825 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f45580541d0 con 0x7f4564099f80 2024-03-28T16:15:30.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.038+0000 7f454bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:30.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.038+0000 7f454bfff640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:30.080 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:15:30.076+0000 ac7e640 -1 osd.2 0 waiting for initial osdmap 2024-03-28T16:15:30.170 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:15:30.165+0000 c497640 -1 osd.3 0 waiting for initial osdmap 2024-03-28T16:15:30.284 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.284+0000 7f4568ba2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:30.284 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.284+0000 7f4568ba2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:30.285 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.284+0000 7f4568ba2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:30.285 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.284+0000 7f4568ba2640 1 -- 172.21.15.7:0/402325825 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4564074a00 con 0x7f4564099f80 2024-03-28T16:15:30.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.319+0000 7f454bfff640 1 -- 172.21.15.7:0/402325825 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+8333 (secure 0 0 0) 0x7f4558020090 con 0x7f4564099f80 2024-03-28T16:15:30.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.319+0000 7f454bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:30.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.319+0000 7f454bfff640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:30.320 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-28T16:15:30.320 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":15,"fsid":"808ff1a1-2b17-4c35-a89a-340dbbf674ee","created":"2024-03-28T16:14:44.591599+0000","modified":"2024-03-28T16:15:29.605924+0000","last_up_change":"2024-03-28T16:15:29.605924+0000","last_in_change":"2024-03-28T16:14:49.757700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a19c6856-78b9-4919-97ce-6940a5af982c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":34918},{"type":"v1","addr":"172.21.15.7:6809","nonce":34918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":34918},{"type":"v1","addr":"172.21.15.7:6811","nonce":34918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":34918},{"type":"v1","addr":"172.21.15.7:6815","nonce":34918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":34918},{"type":"v1","addr":"172.21.15.7:6813","nonce":34918}]},"public_addr":"172.21.15.7:6809/34918","cluster_addr":"172.21.15.7:6811/34918","heartbeat_back_addr":"172.21.15.7:6815/34918","heartbeat_front_addr":"172.21.15.7:6813/34918","state":["exists","up"]},{"osd":1,"uuid":"2afee73d-3369-4fd2-bfc5-36dee610daf2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0473d84c-e864-4373-b9d1-283130278363","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b64fd5e7-cdec-4bd0-b5e2-dfc8d4f6b9ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8f7552db-5651-430f-a826-3c6843bffb99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":35294},{"type":"v1","addr":"172.21.15.83:6819","nonce":35294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":35294},{"type":"v1","addr":"172.21.15.83:6821","nonce":35294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":35294},{"type":"v1","addr":"172.21.15.83:6825","nonce":35294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":35294},{"type":"v1","addr":"172.21.15.83:6823","nonce":35294}]},"public_addr":"172.21.15.83:6819/35294","cluster_addr":"172.21.15.83:6821/35294","heartbeat_back_addr":"172.21.15.83:6825/35294","heartbeat_front_addr":"172.21.15.83:6823/35294","state":["exists","up"]},{"osd":5,"uuid":"2aabdbd2-d870-4d6d-823e-89b008ea3108","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fbbb7557-08d4-48d4-9b88-6bb1d680a87d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"805f54d9-03d5-4b23-aef3-e1d55bb09f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":35295},{"type":"v1","addr":"172.21.15.83:6803","nonce":35295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":35295},{"type":"v1","addr":"172.21.15.83:6805","nonce":35295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":35295},{"type":"v1","addr":"172.21.15.83:6809","nonce":35295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":35295},{"type":"v1","addr":"172.21.15.83:6807","nonce":35295}]},"public_addr":"172.21.15.83:6803/35295","cluster_addr":"172.21.15.83:6805/35295","heartbeat_back_addr":"172.21.15.83:6809/35295","heartbeat_front_addr":"172.21.15.83:6807/35295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:26.831000+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:15:30.322 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.321+0000 7f4568ba2640 10 librados: watch_flush enter 2024-03-28T16:15:30.322 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.321+0000 7f4568ba2640 10 librados: watch_flush exit 2024-03-28T16:15:30.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.321+0000 7f4568ba2640 20 client.4229.objecter shutdown clearing up homeless session... 2024-03-28T16:15:30.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.321+0000 7f4568ba2640 10 client.4229.objecter successfully canceled tick 2024-03-28T16:15:30.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.321+0000 7f4568ba2640 1 -- 172.21.15.7:0/402325825 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f455006c550 msgr2=0x7f455006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:30.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.321+0000 7f4568ba2640 1 --2- 172.21.15.7:0/402325825 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f455006c550 0x7f455006ea10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4554036000 tx=0x7f455403d040 comp rx=0 tx=0).stop 2024-03-28T16:15:30.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.321+0000 7f4568ba2640 10 monclient: shutdown 2024-03-28T16:15:30.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.321+0000 7f4568ba2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:30.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.321+0000 7f4568ba2640 1 -- 172.21.15.7:0/402325825 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4564099f80 msgr2=0x7f4564136e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:30.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.321+0000 7f4568ba2640 1 --2- 172.21.15.7:0/402325825 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4564099f80 0x7f4564136e10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f455800c400 tx=0x7f455800c8c0 comp rx=0 tx=0).stop 2024-03-28T16:15:30.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.322+0000 7f4568ba2640 1 -- 172.21.15.7:0/402325825 shutdown_connections 2024-03-28T16:15:30.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.322+0000 7f4568ba2640 1 --2- 172.21.15.7:0/402325825 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f455006c550 0x7f455006ea10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:30.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.322+0000 7f4568ba2640 1 --2- 172.21.15.7:0/402325825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f456413e550 0x7f45641d96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:30.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.322+0000 7f4568ba2640 1 --2- 172.21.15.7:0/402325825 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4564099f80 0x7f4564136e10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:30.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.322+0000 7f4568ba2640 1 --2- 172.21.15.7:0/402325825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f456406a600 0x7f4564133380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:30.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.322+0000 7f4568ba2640 1 -- 172.21.15.7:0/402325825 >> 172.21.15.7:0/402325825 conn(0x7f45640b6f50 msgr2=0x7f45640b3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:30.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.322+0000 7f4568ba2640 1 -- 172.21.15.7:0/402325825 shutdown_connections 2024-03-28T16:15:30.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.322+0000 7f4568ba2640 1 -- 172.21.15.7:0/402325825 wait complete. 2024-03-28T16:15:30.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:30.322+0000 7f4568ba2640 1 librados: shutdown 2024-03-28T16:15:30.333 DEBUG:teuthology.misc:3 of 8 OSDs are up 2024-03-28T16:15:30.425 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-03-28T16:15:30.421+0000 ac7e640 -1 osd.1 0 waiting for initial osdmap 2024-03-28T16:15:30.645 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:15:30.630+0000 1025c640 -1 osd.2 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T16:15:30.721 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:15:30.707+0000 1025c640 -1 osd.3 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T16:15:30.895 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-03-28T16:15:30.882+0000 1025c640 -1 osd.1 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T16:15:34.756 INFO:tasks.ceph.osd.6.smithi083.stderr:2024-03-28T16:15:34.669+0000 5dc2480 -1 bdev(0x18e9ca40 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 2 2024-03-28T16:15:34.857 INFO:tasks.ceph.osd.6.smithi083.stderr:2024-03-28T16:15:34.756+0000 5dc2480 -1 bdev(0x18e9ca40 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-03-28T16:15:34.897 INFO:tasks.ceph.osd.6.smithi083.stderr:2024-03-28T16:15:34.857+0000 5dc2480 -1 bdev(0x18e9ca40 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 2 2024-03-28T16:15:35.254 INFO:tasks.ceph.osd.6.smithi083.stderr:2024-03-28T16:15:34.897+0000 5dc2480 -1 bdev(0x18e9ca40 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-03-28T16:15:35.416 INFO:tasks.ceph.osd.6.smithi083.stderr:2024-03-28T16:15:35.253+0000 5dc2480 -1 bdev(0x18e9ca40 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-03-28T16:15:35.550 INFO:tasks.ceph.osd.6.smithi083.stderr:2024-03-28T16:15:35.372+0000 5dc2480 -1 bdev(0x18e9ca40 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-03-28T16:15:36.482 INFO:tasks.ceph.osd.6.smithi083.stderr:2024-03-28T16:15:36.477+0000 ac7e640 -1 osd.6 0 waiting for initial osdmap 2024-03-28T16:15:36.930 INFO:tasks.ceph.osd.6.smithi083.stderr:2024-03-28T16:15:36.916+0000 14264640 -1 osd.6 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T16:15:36.937 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:15:36.781+0000 5dc2480 -1 bdev(0x18ecbd00 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-03-28T16:15:36.971 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:15:36.937+0000 5dc2480 -1 bdev(0x18ecbd00 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-03-28T16:15:36.971 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:15:36.959+0000 5dc2480 -1 bdev(0x18ecbd00 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-03-28T16:15:36.982 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:15:36.970+0000 5dc2480 -1 bdev(0x18ecbd00 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 2 2024-03-28T16:15:37.066 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:15:37.030+0000 5dc2480 -1 bdev(0x18ecbd00 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 3 2024-03-28T16:15:37.066 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:15:37.044+0000 5dc2480 -1 bdev(0x18ecbd00 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-03-28T16:15:37.077 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:15:37.055+0000 5dc2480 -1 bdev(0x18ecbd00 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-03-28T16:15:37.091 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:15:37.066+0000 5dc2480 -1 bdev(0x18ecbd00 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-03-28T16:15:37.091 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:15:37.090+0000 5dc2480 -1 bdev(0x18ecbd00 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-03-28T16:15:37.140 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-28T16:15:37.164 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:15:37.094+0000 5dc2480 -1 bdev(0x18ecbd00 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-03-28T16:15:37.164 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:15:37.153+0000 5dc2480 -1 bdev(0x18ecbd00 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-03-28T16:15:37.236 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.235+0000 7f1e722f7640 10 monclient: get_monmap_and_config 2024-03-28T16:15:37.236 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.235+0000 7f1e722f7640 10 monclient: build_initial_monmap 2024-03-28T16:15:37.236 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.236+0000 7f1e722f7640 10 monclient: monmap: 2024-03-28T16:15:37.236 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-28T16:15:37.237 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:37.237 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:15:37.236623+0000 2024-03-28T16:15:37.237 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:15:37.236623+0000 2024-03-28T16:15:37.237 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:37.237 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:37.237 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:37.237 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:37.237 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:37.237 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:37.237 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:37.237 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:37.239 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.236+0000 7f1e722f7640 1 Processor -- start 2024-03-28T16:15:37.239 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.237+0000 7f1e722f7640 1 -- start start 2024-03-28T16:15:37.239 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.237+0000 7f1e722f7640 10 monclient: init 2024-03-28T16:15:37.239 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.237+0000 7f1e722f7640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:37.240 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.237+0000 7f1e722f7640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-03-28T16:15:37.240 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.237+0000 7f1e722f7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1e6c0b3ff0 0x7f1e6c144950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:37.240 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.237+0000 7f1e722f7640 10 monclient(hunting): picked mon.noname-c con 0x7f1e6c0b3ff0 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:37.240 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.237+0000 7f1e722f7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1e6c0b3680 addr v1:172.21.15.83:6789/0 2024-03-28T16:15:37.240 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.237+0000 7f1e722f7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1e6c0a86e0 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:37.240 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.237+0000 7f1e722f7640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e6c0a9170 con 0x7f1e6c0a86e0 2024-03-28T16:15:37.240 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.237+0000 7f1e722f7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e6c0aa4b0 con 0x7f1e6c0b3680 2024-03-28T16:15:37.240 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.237+0000 7f1e722f7640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:37.240 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.237+0000 7f1e722f7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e6c0b43d0 con 0x7f1e6c0b3ff0 2024-03-28T16:15:37.240 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.237+0000 7f1e722f7640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:37.240 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.237+0000 7f1e722f7640 10 monclient(hunting): authenticate will time out at 1404.378490s 2024-03-28T16:15:37.240 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.238+0000 7f1e70894640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f1e6c0a86e0 0x7f1e6c13e550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39036/0 (socket says 172.21.15.7:39036) 2024-03-28T16:15:37.240 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.238+0000 7f1e70894640 1 -- 172.21.15.7:0/35650202 learned_addr learned my addr 172.21.15.7:0/35650202 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-28T16:15:37.241 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.238+0000 7f1e6bfff640 1 --2- 172.21.15.7:0/35650202 >> v2:172.21.15.83:3300/0 conn(0x7f1e6c0b3ff0 0x7f1e6c144950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:37.241 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.238+0000 7f1e6bfff640 10 monclient(hunting): get_auth_request con 0x7f1e6c0b3ff0 auth_method 0 2024-03-28T16:15:37.241 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.238+0000 7f1e6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:37.241 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.238+0000 7f1e6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:37.241 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.238+0000 7f1e6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:37.241 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.239+0000 7f1e6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:37.241 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.239+0000 7f1e6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:37.241 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.239+0000 7f1e6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:37.242 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.241+0000 7f1e6bfff640 10 monclient(hunting): handle_auth_done global_id 4267 payload 274 2024-03-28T16:15:37.242 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.241+0000 7f1e6bfff640 1 -- 172.21.15.7:0/35650202 >> v1:172.21.15.83:6789/0 conn(0x7f1e6c0b3680 legacy=0x7f1e6c144e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:37.242 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.241+0000 7f1e6bfff640 1 -- 172.21.15.7:0/35650202 >> v1:172.21.15.7:6789/0 conn(0x7f1e6c0a86e0 legacy=0x7f1e6c13e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:37.242 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.241+0000 7f1e6bfff640 10 monclient: _finish_hunting 0 2024-03-28T16:15:37.242 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.241+0000 7f1e6bfff640 1 monclient: found mon.noname-c 2024-03-28T16:15:37.242 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.241+0000 7f1e6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:37.242 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.241+0000 7f1e6bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3300/0 2024-03-28T16:15:37.242 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.241+0000 7f1e6bfff640 1 -- 172.21.15.7:0/35650202 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6c0a8c50 con 0x7f1e6c0b3ff0 2024-03-28T16:15:37.243 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.243+0000 7f1e6bfff640 1 --2- 172.21.15.7:0/35650202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e6c0b3ff0 0x7f1e6c144950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c0097a0 tx=0x7f1e5c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=828833946dc5f637 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:37.249 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.248+0000 7f1e6affd640 1 -- 172.21.15.7:0/35650202 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c031830 con 0x7f1e6c0b3ff0 2024-03-28T16:15:37.249 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.248+0000 7f1e6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:37.249 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.248+0000 7f1e6affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:37.249 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.248+0000 7f1e6affd640 10 monclient: dump: 2024-03-28T16:15:37.249 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:37.249 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:37.249 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:37.249 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:37.249 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:37.249 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:37.249 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:37.250 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:37.250 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:37.250 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:37.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.248+0000 7f1e6affd640 10 monclient: _finish_auth 0 2024-03-28T16:15:37.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.248+0000 7f1e6affd640 10 monclient: _check_auth_tickets 2024-03-28T16:15:37.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.248+0000 7f1e6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:37.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.248+0000 7f1e722f7640 5 monclient: authenticate success, global_id 4267 2024-03-28T16:15:37.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.248+0000 7f1e722f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:37.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.248+0000 7f1e6affd640 1 -- 172.21.15.7:0/35650202 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e5c031990 con 0x7f1e6c0b3ff0 2024-03-28T16:15:37.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.248+0000 7f1e6affd640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:37.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.248+0000 7f1e6affd640 1 -- 172.21.15.7:0/35650202 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c031d20 con 0x7f1e6c0b3ff0 2024-03-28T16:15:37.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.248+0000 7f1e722f7640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:37.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.248+0000 7f1e722f7640 4 set_mon_vals no callback set 2024-03-28T16:15:37.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.248+0000 7f1e6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:37.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.248+0000 7f1e6affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:37.251 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.248+0000 7f1e6affd640 10 monclient: dump: 2024-03-28T16:15:37.251 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:37.251 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:37.251 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:37.251 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:37.251 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:37.251 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:37.251 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:37.251 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:37.251 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:37.251 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:37.251 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.248+0000 7f1e722f7640 10 monclient: shutdown 2024-03-28T16:15:37.251 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.248+0000 7f1e722f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:37.251 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.248+0000 7f1e722f7640 1 -- 172.21.15.7:0/35650202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e6c0b3ff0 msgr2=0x7f1e6c144950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:37.252 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.248+0000 7f1e722f7640 1 --2- 172.21.15.7:0/35650202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e6c0b3ff0 0x7f1e6c144950 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c0097a0 tx=0x7f1e5c02b8d0 comp rx=0 tx=0).stop 2024-03-28T16:15:37.252 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.249+0000 7f1e722f7640 1 -- 172.21.15.7:0/35650202 shutdown_connections 2024-03-28T16:15:37.252 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.249+0000 7f1e722f7640 1 --2- 172.21.15.7:0/35650202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e6c0b3ff0 0x7f1e6c144950 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:37.252 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.249+0000 7f1e722f7640 1 -- 172.21.15.7:0/35650202 >> 172.21.15.7:0/35650202 conn(0x7f1e6c0735d0 msgr2=0x7f1e6c0739e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:37.252 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.249+0000 7f1e722f7640 1 -- 172.21.15.7:0/35650202 shutdown_connections 2024-03-28T16:15:37.252 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.249+0000 7f1e722f7640 1 -- 172.21.15.7:0/35650202 wait complete. 2024-03-28T16:15:37.255 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.249+0000 7f1e722f7640 10 monclient: build_initial_monmap 2024-03-28T16:15:37.255 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.249+0000 7f1e722f7640 10 monclient: monmap: 2024-03-28T16:15:37.255 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-28T16:15:37.255 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:37.255 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:37.255 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-28T16:15:37.256 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:37.256 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:37.256 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:37.256 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:37.256 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:37.256 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:37.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 1 librados: starting msgr at 2024-03-28T16:15:37.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 1 librados: starting objecter 2024-03-28T16:15:37.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 1 Processor -- start 2024-03-28T16:15:37.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 1 -- start start 2024-03-28T16:15:37.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 1 librados: setting wanted keys 2024-03-28T16:15:37.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 1 librados: calling monclient init 2024-03-28T16:15:37.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 10 monclient: init 2024-03-28T16:15:37.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:37.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-28T16:15:37.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1e6c0a86e0 0x7f1e6c144010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:37.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 10 monclient(hunting): picked mon.noname-c con 0x7f1e6c0a86e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:37.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e6c0b3ff0 0x7f1e6c133170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:37.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 10 monclient(hunting): picked mon.noname-b con 0x7f1e6c0b3ff0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:37.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e6c138620 0x7f1e6c1336b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:37.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 10 monclient(hunting): picked mon.noname-a con 0x7f1e6c138620 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:37.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:37.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6c141210 con 0x7f1e6c138620 2024-03-28T16:15:37.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:37.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6c133bf0 con 0x7f1e6c0b3ff0 2024-03-28T16:15:37.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:37.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6c140570 con 0x7f1e6c0a86e0 2024-03-28T16:15:37.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:37.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.250+0000 7f1e722f7640 10 monclient(hunting): authenticate will time out at 1404.391774s 2024-03-28T16:15:37.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.251+0000 7f1e6bfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1e6c0a86e0 0x7f1e6c144010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:37.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.251+0000 7f1e70894640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e6c138620 0x7f1e6c1336b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:37.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.251+0000 7f1e6b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e6c0b3ff0 0x7f1e6c133170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:37.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.251+0000 7f1e6bfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1e6c0a86e0 0x7f1e6c144010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:39130/0 (socket says 172.21.15.7:39130) 2024-03-28T16:15:37.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.251+0000 7f1e6bfff640 1 -- 172.21.15.7:0/3415575769 learned_addr learned my addr 172.21.15.7:0/3415575769 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:37.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.251+0000 7f1e6bfff640 10 monclient(hunting): get_auth_request con 0x7f1e6c0a86e0 auth_method 0 2024-03-28T16:15:37.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.251+0000 7f1e6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:37.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.251+0000 7f1e6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:37.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.251+0000 7f1e6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:37.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.251+0000 7f1e6b7fe640 10 monclient(hunting): get_auth_request con 0x7f1e6c0b3ff0 auth_method 0 2024-03-28T16:15:37.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.251+0000 7f1e6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:37.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.251+0000 7f1e6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:37.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.251+0000 7f1e6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:37.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.252+0000 7f1e70894640 10 monclient(hunting): get_auth_request con 0x7f1e6c138620 auth_method 0 2024-03-28T16:15:37.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.252+0000 7f1e70894640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:37.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.252+0000 7f1e70894640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:37.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.252+0000 7f1e70894640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:37.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.252+0000 7f1e6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:37.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.252+0000 7f1e6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:37.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.252+0000 7f1e6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:37.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.252+0000 7f1e6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:37.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.252+0000 7f1e6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:37.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.252+0000 7f1e6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:37.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.252+0000 7f1e70894640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:37.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.252+0000 7f1e70894640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:37.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.252+0000 7f1e70894640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:37.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.255+0000 7f1e6b7fe640 10 monclient(hunting): handle_auth_done global_id 4270 payload 824 2024-03-28T16:15:37.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.255+0000 7f1e6b7fe640 1 -- 172.21.15.7:0/3415575769 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1e6c0a86e0 msgr2=0x7f1e6c144010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:37.260 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.255+0000 7f1e6b7fe640 1 --2- 172.21.15.7:0/3415575769 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1e6c0a86e0 0x7f1e6c144010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:37.260 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.255+0000 7f1e6b7fe640 1 -- 172.21.15.7:0/3415575769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e6c138620 msgr2=0x7f1e6c1336b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:37.260 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.255+0000 7f1e6b7fe640 1 --2- 172.21.15.7:0/3415575769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e6c138620 0x7f1e6c1336b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:37.260 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.255+0000 7f1e6b7fe640 10 monclient: _finish_hunting 0 2024-03-28T16:15:37.260 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.255+0000 7f1e6b7fe640 1 monclient: found mon.noname-b 2024-03-28T16:15:37.260 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.255+0000 7f1e6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:37.260 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.255+0000 7f1e6b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:15:37.260 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.255+0000 7f1e6b7fe640 1 -- 172.21.15.7:0/3415575769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e5c009400 con 0x7f1e6c0b3ff0 2024-03-28T16:15:37.262 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.257+0000 7f1e6b7fe640 1 --2- 172.21.15.7:0/3415575769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e6c0b3ff0 0x7f1e6c133170 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1e6000c410 tx=0x7f1e6000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:37.262 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.257+0000 7f1e68ff9640 10 client.?.objecter ms_handle_connect 0x7f1e6c0b3ff0 2024-03-28T16:15:37.262 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.257+0000 7f1e68ff9640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:37.262 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e68ff9640 1 -- 172.21.15.7:0/3415575769 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e600035f0 con 0x7f1e6c0b3ff0 2024-03-28T16:15:37.262 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:37.262 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e68ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:37.262 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e68ff9640 10 monclient: dump: 2024-03-28T16:15:37.262 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:37.262 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:37.262 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:37.262 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:37.263 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:37.263 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:37.263 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:37.263 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:37.263 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:37.263 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:37.263 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e68ff9640 10 monclient: _finish_auth 0 2024-03-28T16:15:37.263 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e68ff9640 10 monclient: _check_auth_tickets 2024-03-28T16:15:37.263 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:37.263 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e68ff9640 1 -- 172.21.15.7:0/3415575769 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e60003750 con 0x7f1e6c0b3ff0 2024-03-28T16:15:37.263 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e722f7640 5 monclient: authenticate success, global_id 4270 2024-03-28T16:15:37.263 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e722f7640 10 monclient: _renew_subs 2024-03-28T16:15:37.263 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e722f7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:37.263 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e722f7640 1 -- 172.21.15.7:0/3415575769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e6c141760 con 0x7f1e6c0b3ff0 2024-03-28T16:15:37.263 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e68ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:37.264 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e68ff9640 1 -- 172.21.15.7:0/3415575769 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e60003ae0 con 0x7f1e6c0b3ff0 2024-03-28T16:15:37.264 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:37.264 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e68ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:37.264 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e68ff9640 10 monclient: dump: 2024-03-28T16:15:37.264 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:37.264 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:37.264 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:37.264 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:37.264 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:37.264 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:37.264 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:37.264 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:37.264 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:37.264 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:37.264 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e6a7fc640 4 set_mon_vals no callback set 2024-03-28T16:15:37.264 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e722f7640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:37.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e722f7640 10 monclient: _renew_subs 2024-03-28T16:15:37.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e722f7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:37.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e722f7640 1 -- 172.21.15.7:0/3415575769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e6c142340 con 0x7f1e6c0b3ff0 2024-03-28T16:15:37.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.260+0000 7f1e722f7640 1 librados: init done 2024-03-28T16:15:37.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.261+0000 7f1e722f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:37.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.261+0000 7f1e722f7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:37.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.261+0000 7f1e722f7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:37.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.261+0000 7f1e722f7640 1 -- 172.21.15.7:0/3415575769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e6c0670c0 con 0x7f1e6c0b3ff0 2024-03-28T16:15:37.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.279+0000 7f1e68ff9640 1 -- 172.21.15.7:0/3415575769 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 6) v1 ==== 89081+0+0 (secure 0 0 0) 0x7f1e60003c40 con 0x7f1e6c0b3ff0 2024-03-28T16:15:37.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.279+0000 7f1e68ff9640 1 --2- 172.21.15.7:0/3415575769 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f1e5806c550 0x7f1e5806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:37.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.279+0000 7f1e6bfff640 1 --2- 172.21.15.7:0/3415575769 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f1e5806c550 0x7f1e5806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:37.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.279+0000 7f1e6bfff640 10 monclient: get_auth_request con 0x7f1e5806c550 auth_method 0 2024-03-28T16:15:37.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.280+0000 7f1e6bfff640 1 --2- 172.21.15.7:0/3415575769 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f1e5806c550 0x7f1e5806ea10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c02b910 tx=0x7f1e5c03c040 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:37.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.280+0000 7f1e68ff9640 10 client.4270.objecter ms_handle_connect 0x7f1e5806c550 2024-03-28T16:15:37.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.282+0000 7f1e68ff9640 1 -- 172.21.15.7:0/3415575769 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 4297+0+0 (secure 0 0 0) 0x7f1e600580f0 con 0x7f1e6c0b3ff0 2024-03-28T16:15:37.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.282+0000 7f1e68ff9640 10 client.4270.objecter ms_dispatch 0x7f1e6c000b90 osd_map(20..20 src has 1..20) v4 2024-03-28T16:15:37.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.282+0000 7f1e68ff9640 3 client.4270.objecter handle_osd_map got epochs [20,20] > 0 2024-03-28T16:15:37.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.282+0000 7f1e68ff9640 3 client.4270.objecter handle_osd_map decoding full epoch 20 2024-03-28T16:15:37.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.282+0000 7f1e68ff9640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-28T16:15:37.525 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.524+0000 7f1e68ff9640 1 -- 172.21.15.7:0/3415575769 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1e60054430 con 0x7f1e6c0b3ff0 2024-03-28T16:15:37.525 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.524+0000 7f1e68ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:37.525 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.524+0000 7f1e68ff9640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:37.579 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:15:37.206+0000 5dc2480 -1 bdev(0x18ecbd00 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-03-28T16:15:37.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.726+0000 7f1e722f7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:37.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.726+0000 7f1e722f7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:37.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.726+0000 7f1e722f7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:37.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.726+0000 7f1e722f7640 1 -- 172.21.15.7:0/3415575769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e6c141350 con 0x7f1e6c0b3ff0 2024-03-28T16:15:37.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.939+0000 7f1e68ff9640 1 -- 172.21.15.7:0/3415575769 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+11873 (secure 0 0 0) 0x7f1e60017090 con 0x7f1e6c0b3ff0 2024-03-28T16:15:37.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.939+0000 7f1e68ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:37.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.939+0000 7f1e68ff9640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:37.940 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-28T16:15:37.940 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":21,"fsid":"808ff1a1-2b17-4c35-a89a-340dbbf674ee","created":"2024-03-28T16:14:44.591599+0000","modified":"2024-03-28T16:15:37.653165+0000","last_up_change":"2024-03-28T16:15:37.653165+0000","last_in_change":"2024-03-28T16:14:49.757700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T16:15:31.373047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.690000057220459,"score_stable":7.690000057220459,"optimal_score":0.12999999523162842,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a19c6856-78b9-4919-97ce-6940a5af982c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":34918},{"type":"v1","addr":"172.21.15.7:6809","nonce":34918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":34918},{"type":"v1","addr":"172.21.15.7:6811","nonce":34918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":34918},{"type":"v1","addr":"172.21.15.7:6815","nonce":34918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":34918},{"type":"v1","addr":"172.21.15.7:6813","nonce":34918}]},"public_addr":"172.21.15.7:6809/34918","cluster_addr":"172.21.15.7:6811/34918","heartbeat_back_addr":"172.21.15.7:6815/34918","heartbeat_front_addr":"172.21.15.7:6813/34918","state":["exists","up"]},{"osd":1,"uuid":"2afee73d-3369-4fd2-bfc5-36dee610daf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":34919},{"type":"v1","addr":"172.21.15.7:6801","nonce":34919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":34919},{"type":"v1","addr":"172.21.15.7:6803","nonce":34919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":34919},{"type":"v1","addr":"172.21.15.7:6807","nonce":34919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":34919},{"type":"v1","addr":"172.21.15.7:6805","nonce":34919}]},"public_addr":"172.21.15.7:6801/34919","cluster_addr":"172.21.15.7:6803/34919","heartbeat_back_addr":"172.21.15.7:6807/34919","heartbeat_front_addr":"172.21.15.7:6805/34919","state":["exists","up"]},{"osd":2,"uuid":"0473d84c-e864-4373-b9d1-283130278363","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":34920},{"type":"v1","addr":"172.21.15.7:6825","nonce":34920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":34920},{"type":"v1","addr":"172.21.15.7:6827","nonce":34920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":34920},{"type":"v1","addr":"172.21.15.7:6831","nonce":34920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":34920},{"type":"v1","addr":"172.21.15.7:6829","nonce":34920}]},"public_addr":"172.21.15.7:6825/34920","cluster_addr":"172.21.15.7:6827/34920","heartbeat_back_addr":"172.21.15.7:6831/34920","heartbeat_front_addr":"172.21.15.7:6829/34920","state":["exists","up"]},{"osd":3,"uuid":"b64fd5e7-cdec-4bd0-b5e2-dfc8d4f6b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":34921},{"type":"v1","addr":"172.21.15.7:6817","nonce":34921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":34921},{"type":"v1","addr":"172.21.15.7:6819","nonce":34921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":34921},{"type":"v1","addr":"172.21.15.7:6823","nonce":34921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":34921},{"type":"v1","addr":"172.21.15.7:6821","nonce":34921}]},"public_addr":"172.21.15.7:6817/34921","cluster_addr":"172.21.15.7:6819/34921","heartbeat_back_addr":"172.21.15.7:6823/34921","heartbeat_front_addr":"172.21.15.7:6821/34921","state":["exists","up"]},{"osd":4,"uuid":"8f7552db-5651-430f-a826-3c6843bffb99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":35294},{"type":"v1","addr":"172.21.15.83:6819","nonce":35294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":35294},{"type":"v1","addr":"172.21.15.83:6821","nonce":35294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":35294},{"type":"v1","addr":"172.21.15.83:6825","nonce":35294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":35294},{"type":"v1","addr":"172.21.15.83:6823","nonce":35294}]},"public_addr":"172.21.15.83:6819/35294","cluster_addr":"172.21.15.83:6821/35294","heartbeat_back_addr":"172.21.15.83:6825/35294","heartbeat_front_addr":"172.21.15.83:6823/35294","state":["exists","up"]},{"osd":5,"uuid":"2aabdbd2-d870-4d6d-823e-89b008ea3108","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fbbb7557-08d4-48d4-9b88-6bb1d680a87d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":35297},{"type":"v1","addr":"172.21.15.83:6811","nonce":35297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":35297},{"type":"v1","addr":"172.21.15.83:6813","nonce":35297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":35297},{"type":"v1","addr":"172.21.15.83:6817","nonce":35297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":35297},{"type":"v1","addr":"172.21.15.83:6815","nonce":35297}]},"public_addr":"172.21.15.83:6811/35297","cluster_addr":"172.21.15.83:6813/35297","heartbeat_back_addr":"172.21.15.83:6817/35297","heartbeat_front_addr":"172.21.15.83:6815/35297","state":["exists","up"]},{"osd":7,"uuid":"805f54d9-03d5-4b23-aef3-e1d55bb09f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":35295},{"type":"v1","addr":"172.21.15.83:6803","nonce":35295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":35295},{"type":"v1","addr":"172.21.15.83:6805","nonce":35295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":35295},{"type":"v1","addr":"172.21.15.83:6809","nonce":35295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":35295},{"type":"v1","addr":"172.21.15.83:6807","nonce":35295}]},"public_addr":"172.21.15.83:6803/35295","cluster_addr":"172.21.15.83:6805/35295","heartbeat_back_addr":"172.21.15.83:6809/35295","heartbeat_front_addr":"172.21.15.83:6807/35295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:27.198013+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:28.569762+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:28.085669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:28.165751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:26.831000+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:26.424892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:15:37.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.941+0000 7f1e722f7640 10 librados: watch_flush enter 2024-03-28T16:15:37.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.941+0000 7f1e722f7640 10 librados: watch_flush exit 2024-03-28T16:15:37.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.941+0000 7f1e722f7640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-28T16:15:37.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.941+0000 7f1e722f7640 10 client.4270.objecter successfully canceled tick 2024-03-28T16:15:37.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.941+0000 7f1e722f7640 1 -- 172.21.15.7:0/3415575769 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f1e5806c550 msgr2=0x7f1e5806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:37.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.941+0000 7f1e722f7640 1 --2- 172.21.15.7:0/3415575769 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f1e5806c550 0x7f1e5806ea10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c02b910 tx=0x7f1e5c03c040 comp rx=0 tx=0).stop 2024-03-28T16:15:37.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.941+0000 7f1e722f7640 10 monclient: shutdown 2024-03-28T16:15:37.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.941+0000 7f1e722f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:37.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.941+0000 7f1e722f7640 1 -- 172.21.15.7:0/3415575769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e6c0b3ff0 msgr2=0x7f1e6c133170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:37.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.941+0000 7f1e722f7640 1 --2- 172.21.15.7:0/3415575769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e6c0b3ff0 0x7f1e6c133170 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1e6000c410 tx=0x7f1e6000c8d0 comp rx=0 tx=0).stop 2024-03-28T16:15:37.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.941+0000 7f1e722f7640 1 -- 172.21.15.7:0/3415575769 shutdown_connections 2024-03-28T16:15:37.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.941+0000 7f1e722f7640 1 --2- 172.21.15.7:0/3415575769 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f1e5806c550 0x7f1e5806ea10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:37.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.941+0000 7f1e722f7640 1 --2- 172.21.15.7:0/3415575769 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1e6c138620 0x7f1e6c1336b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:37.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.941+0000 7f1e722f7640 1 --2- 172.21.15.7:0/3415575769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e6c0b3ff0 0x7f1e6c133170 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:37.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.941+0000 7f1e722f7640 1 --2- 172.21.15.7:0/3415575769 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1e6c0a86e0 0x7f1e6c144010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:37.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.941+0000 7f1e722f7640 1 -- 172.21.15.7:0/3415575769 >> 172.21.15.7:0/3415575769 conn(0x7f1e6c0735d0 msgr2=0x7f1e6c0b76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:37.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.942+0000 7f1e722f7640 1 -- 172.21.15.7:0/3415575769 shutdown_connections 2024-03-28T16:15:37.944 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.942+0000 7f1e722f7640 1 -- 172.21.15.7:0/3415575769 wait complete. 2024-03-28T16:15:37.944 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:37.942+0000 7f1e722f7640 1 librados: shutdown 2024-03-28T16:15:37.952 DEBUG:teuthology.misc:7 of 8 OSDs are up 2024-03-28T16:15:38.623 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:15:38.619+0000 c497640 -1 osd.5 0 waiting for initial osdmap 2024-03-28T16:15:39.147 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:15:39.130+0000 14264640 -1 osd.5 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T16:15:44.759 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-28T16:15:44.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.849+0000 7ff0cc0d4640 10 monclient: get_monmap_and_config 2024-03-28T16:15:44.850 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.849+0000 7ff0cc0d4640 10 monclient: build_initial_monmap 2024-03-28T16:15:44.850 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.849+0000 7ff0cc0d4640 10 monclient: monmap: 2024-03-28T16:15:44.850 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-28T16:15:44.850 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:44.850 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:15:44.850495+0000 2024-03-28T16:15:44.850 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:15:44.850495+0000 2024-03-28T16:15:44.850 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:44.850 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:44.850 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:44.850 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:44.850 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:44.851 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:44.851 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:44.851 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:44.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.850+0000 7ff0cc0d4640 1 Processor -- start 2024-03-28T16:15:44.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.850+0000 7ff0cc0d4640 1 -- start start 2024-03-28T16:15:44.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.850+0000 7ff0cc0d4640 10 monclient: init 2024-03-28T16:15:44.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.851+0000 7ff0cc0d4640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:44.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.851+0000 7ff0cc0d4640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-03-28T16:15:44.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.851+0000 7ff0cc0d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff0c40719a0 addr v1:172.21.15.83:6789/0 2024-03-28T16:15:44.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.851+0000 7ff0cc0d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0c413eeb0 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:44.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.851+0000 7ff0cc0d4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ff0c4141e50 0x7ff0c414ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:44.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.851+0000 7ff0cc0d4640 10 monclient(hunting): picked mon.noname-c con 0x7ff0c4141e50 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:44.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.851+0000 7ff0cc0d4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c40aa3f0 con 0x7ff0c413eeb0 2024-03-28T16:15:44.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.851+0000 7ff0cc0d4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c40aa6d0 con 0x7ff0c40719a0 2024-03-28T16:15:44.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.851+0000 7ff0cc0d4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:44.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.851+0000 7ff0cc0d4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c4071d80 con 0x7ff0c4141e50 2024-03-28T16:15:44.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.851+0000 7ff0cc0d4640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:44.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.851+0000 7ff0cc0d4640 10 monclient(hunting): authenticate will time out at 1411.992142s 2024-03-28T16:15:44.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.851+0000 7ff0c966f640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff0c413eeb0 0x7ff0c4141740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39050/0 (socket says 172.21.15.7:39050) 2024-03-28T16:15:44.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.851+0000 7ff0c966f640 1 -- 172.21.15.7:0/3821410287 learned_addr learned my addr 172.21.15.7:0/3821410287 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-28T16:15:44.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.851+0000 7ff0ca671640 1 --2- 172.21.15.7:0/3821410287 >> v2:172.21.15.83:3300/0 conn(0x7ff0c4141e50 0x7ff0c414ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:44.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.852+0000 7ff0ca671640 10 monclient(hunting): get_auth_request con 0x7ff0c4141e50 auth_method 0 2024-03-28T16:15:44.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.852+0000 7ff0ca671640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:44.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.852+0000 7ff0ca671640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:44.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.852+0000 7ff0ca671640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:44.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.852+0000 7ff0ca671640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:44.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.852+0000 7ff0ca671640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:44.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.853+0000 7ff0ca671640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:44.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.854+0000 7ff0c8e6e640 1 -- 172.21.15.7:0/3821410287 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711349333 0 0) 0x7ff0c40aa3f0 con 0x7ff0c413eeb0 2024-03-28T16:15:44.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.854+0000 7ff0c8e6e640 10 monclient(hunting): _negotiate 2024-03-28T16:15:44.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.854+0000 7ff0c8e6e640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:44.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.854+0000 7ff0c8e6e640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:44.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.854+0000 7ff0c8e6e640 10 monclient(hunting): my global_id is 4275 2024-03-28T16:15:44.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.854+0000 7ff0c8e6e640 1 -- 172.21.15.7:0/3821410287 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0b0003520 con 0x7ff0c413eeb0 2024-03-28T16:15:44.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.854+0000 7ff0ca671640 10 monclient(hunting): handle_auth_done global_id 4276 payload 274 2024-03-28T16:15:44.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.854+0000 7ff0ca671640 1 -- 172.21.15.7:0/3821410287 >> v1:172.21.15.83:6789/0 conn(0x7ff0c40719a0 legacy=0x7ff0c413e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:44.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.854+0000 7ff0ca671640 1 -- 172.21.15.7:0/3821410287 >> v1:172.21.15.7:6789/0 conn(0x7ff0c413eeb0 legacy=0x7ff0c4141740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:44.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.854+0000 7ff0ca671640 10 monclient: _finish_hunting 0 2024-03-28T16:15:44.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.854+0000 7ff0ca671640 1 monclient: found mon.noname-c 2024-03-28T16:15:44.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.854+0000 7ff0ca671640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:44.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.854+0000 7ff0ca671640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3300/0 2024-03-28T16:15:44.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.854+0000 7ff0ca671640 1 -- 172.21.15.7:0/3821410287 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c40a9290 con 0x7ff0c4141e50 2024-03-28T16:15:44.857 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.857+0000 7ff0ca671640 1 --2- 172.21.15.7:0/3821410287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0c4141e50 0x7ff0c414ce20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff0c40b3350 tx=0x7ff0bc02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0c324515400a481 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:44.862 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0c8e6e640 1 -- 172.21.15.7:0/3821410287 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc02cbc0 con 0x7ff0c4141e50 2024-03-28T16:15:44.862 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0c8e6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:44.862 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0c8e6e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:44.862 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0c8e6e640 10 monclient: dump: 2024-03-28T16:15:44.862 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:44.862 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:44.862 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:44.862 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:44.862 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:44.862 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:44.862 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:44.862 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:44.862 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:44.863 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:44.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0c8e6e640 10 monclient: _finish_auth 0 2024-03-28T16:15:44.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0c8e6e640 10 monclient: _check_auth_tickets 2024-03-28T16:15:44.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0c8e6e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:44.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0cc0d4640 5 monclient: authenticate success, global_id 4276 2024-03-28T16:15:44.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0cc0d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:44.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0c8e6e640 1 -- 172.21.15.7:0/3821410287 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0bc02cd20 con 0x7ff0c4141e50 2024-03-28T16:15:44.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0c8e6e640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:44.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0cc0d4640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:44.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0cc0d4640 4 set_mon_vals no callback set 2024-03-28T16:15:44.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0c8e6e640 1 -- 172.21.15.7:0/3821410287 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc034660 con 0x7ff0c4141e50 2024-03-28T16:15:44.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0c8e6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:44.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0c8e6e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:44.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0c8e6e640 10 monclient: dump: 2024-03-28T16:15:44.863 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:44.864 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:44.864 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:44.864 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:44.864 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:44.864 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:44.864 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:44.864 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:44.864 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:44.864 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:44.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0cc0d4640 10 monclient: shutdown 2024-03-28T16:15:44.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0cc0d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:44.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0cc0d4640 1 -- 172.21.15.7:0/3821410287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0c4141e50 msgr2=0x7ff0c414ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:44.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0cc0d4640 1 --2- 172.21.15.7:0/3821410287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0c4141e50 0x7ff0c414ce20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff0c40b3350 tx=0x7ff0bc02be10 comp rx=0 tx=0).stop 2024-03-28T16:15:44.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0cc0d4640 1 -- 172.21.15.7:0/3821410287 shutdown_connections 2024-03-28T16:15:44.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0cc0d4640 1 --2- 172.21.15.7:0/3821410287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0c4141e50 0x7ff0c414ce20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:44.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.861+0000 7ff0cc0d4640 1 -- 172.21.15.7:0/3821410287 >> 172.21.15.7:0/3821410287 conn(0x7ff0c40b72b0 msgr2=0x7ff0c40b76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:44.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.862+0000 7ff0cc0d4640 1 -- 172.21.15.7:0/3821410287 shutdown_connections 2024-03-28T16:15:44.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.862+0000 7ff0cc0d4640 1 -- 172.21.15.7:0/3821410287 wait complete. 2024-03-28T16:15:44.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.862+0000 7ff0cc0d4640 10 monclient: build_initial_monmap 2024-03-28T16:15:44.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.862+0000 7ff0cc0d4640 10 monclient: monmap: 2024-03-28T16:15:44.868 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-28T16:15:44.868 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:44.868 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:44.868 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-28T16:15:44.868 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:44.868 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:44.868 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:44.868 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:44.869 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:44.869 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:44.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.862+0000 7ff0cc0d4640 1 librados: starting msgr at 2024-03-28T16:15:44.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.862+0000 7ff0cc0d4640 1 librados: starting objecter 2024-03-28T16:15:44.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.863+0000 7ff0cc0d4640 1 Processor -- start 2024-03-28T16:15:44.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.863+0000 7ff0cc0d4640 1 -- start start 2024-03-28T16:15:44.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.863+0000 7ff0cc0d4640 1 librados: setting wanted keys 2024-03-28T16:15:44.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.863+0000 7ff0cc0d4640 1 librados: calling monclient init 2024-03-28T16:15:44.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.863+0000 7ff0cc0d4640 10 monclient: init 2024-03-28T16:15:44.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.863+0000 7ff0cc0d4640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:44.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.863+0000 7ff0cc0d4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-28T16:15:44.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.863+0000 7ff0cc0d4640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff0c40719a0 0x7ff0c4140170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:44.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.863+0000 7ff0cc0d4640 10 monclient(hunting): picked mon.noname-c con 0x7ff0c40719a0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:44.870 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.863+0000 7ff0cc0d4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0c413eeb0 0x7ff0c4143a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:44.870 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.863+0000 7ff0cc0d4640 10 monclient(hunting): picked mon.noname-a con 0x7ff0c413eeb0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:44.870 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.863+0000 7ff0cc0d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0c4141e50 0x7ff0c4143fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:44.870 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.863+0000 7ff0cc0d4640 10 monclient(hunting): picked mon.noname-b con 0x7ff0c4141e50 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:44.870 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.863+0000 7ff0cc0d4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:44.870 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.863+0000 7ff0cc0d4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c4143160 con 0x7ff0c413eeb0 2024-03-28T16:15:44.870 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.863+0000 7ff0cc0d4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:44.870 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.863+0000 7ff0cc0d4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c41406b0 con 0x7ff0c4141e50 2024-03-28T16:15:44.870 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.863+0000 7ff0cc0d4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:44.870 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.863+0000 7ff0cc0d4640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c4140b60 con 0x7ff0c40719a0 2024-03-28T16:15:44.870 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.863+0000 7ff0cc0d4640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:44.870 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.863+0000 7ff0cc0d4640 10 monclient(hunting): authenticate will time out at 1412.004447s 2024-03-28T16:15:44.870 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0ca671640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0c4141e50 0x7ff0c4143fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:44.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0c966f640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0c413eeb0 0x7ff0c4143a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:44.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0c9e70640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff0c40719a0 0x7ff0c4140170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:44.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0ca671640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0c4141e50 0x7ff0c4143fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:33382/0 (socket says 172.21.15.7:33382) 2024-03-28T16:15:44.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0ca671640 1 -- 172.21.15.7:0/1747986334 learned_addr learned my addr 172.21.15.7:0/1747986334 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:44.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0ca671640 10 monclient(hunting): get_auth_request con 0x7ff0c4141e50 auth_method 0 2024-03-28T16:15:44.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0ca671640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:44.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0c9e70640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff0c40719a0 0x7ff0c4140170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:39132/0 (socket says 172.21.15.7:39132) 2024-03-28T16:15:44.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0ca671640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:44.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0ca671640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:44.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0c9e70640 10 monclient(hunting): get_auth_request con 0x7ff0c40719a0 auth_method 0 2024-03-28T16:15:44.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0c9e70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:44.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0c9e70640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:44.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0c9e70640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:44.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0c966f640 10 monclient(hunting): get_auth_request con 0x7ff0c413eeb0 auth_method 0 2024-03-28T16:15:44.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0c966f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:44.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0c966f640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:44.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0c966f640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:44.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0ca671640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:44.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0ca671640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:44.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0ca671640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:44.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0c9e70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:44.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0c9e70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:44.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.864+0000 7ff0c9e70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:44.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.865+0000 7ff0c966f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:44.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.865+0000 7ff0c966f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:44.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.865+0000 7ff0c966f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:44.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.867+0000 7ff0ca671640 10 monclient(hunting): handle_auth_done global_id 4279 payload 824 2024-03-28T16:15:44.873 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.867+0000 7ff0ca671640 1 -- 172.21.15.7:0/1747986334 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff0c40719a0 msgr2=0x7ff0c4140170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:44.873 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.867+0000 7ff0ca671640 1 --2- 172.21.15.7:0/1747986334 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff0c40719a0 0x7ff0c4140170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:44.873 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.867+0000 7ff0ca671640 1 -- 172.21.15.7:0/1747986334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0c413eeb0 msgr2=0x7ff0c4143a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:44.873 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.867+0000 7ff0ca671640 1 --2- 172.21.15.7:0/1747986334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0c413eeb0 0x7ff0c4143a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:44.873 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.867+0000 7ff0ca671640 10 monclient: _finish_hunting 0 2024-03-28T16:15:44.873 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.867+0000 7ff0ca671640 1 monclient: found mon.noname-b 2024-03-28T16:15:44.873 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.867+0000 7ff0ca671640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:44.873 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.867+0000 7ff0ca671640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:15:44.873 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.867+0000 7ff0ca671640 1 -- 172.21.15.7:0/1747986334 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0bc009a70 con 0x7ff0c4141e50 2024-03-28T16:15:44.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.869+0000 7ff0ca671640 1 --2- 172.21.15.7:0/1747986334 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0c4141e50 0x7ff0c4143fd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc009e80 tx=0x7ff0bc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:44.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.869+0000 7ff0ba7fc640 10 client.?.objecter ms_handle_connect 0x7ff0c4141e50 2024-03-28T16:15:44.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.869+0000 7ff0ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:44.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0ba7fc640 1 -- 172.21.15.7:0/1747986334 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc03e610 con 0x7ff0c4141e50 2024-03-28T16:15:44.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:44.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0ba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:44.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0ba7fc640 10 monclient: dump: 2024-03-28T16:15:44.875 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:44.875 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:44.875 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:44.875 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:44.875 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:44.876 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:44.876 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:44.876 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:44.876 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:44.876 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:44.876 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0ba7fc640 10 monclient: _finish_auth 0 2024-03-28T16:15:44.876 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0ba7fc640 10 monclient: _check_auth_tickets 2024-03-28T16:15:44.876 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:44.876 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0cc0d4640 5 monclient: authenticate success, global_id 4279 2024-03-28T16:15:44.876 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0ba7fc640 1 -- 172.21.15.7:0/1747986334 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0bc02ce90 con 0x7ff0c4141e50 2024-03-28T16:15:44.876 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:44.876 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0ba7fc640 1 -- 172.21.15.7:0/1747986334 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc03e990 con 0x7ff0c4141e50 2024-03-28T16:15:44.876 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:44.877 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0ba7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:44.877 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0ba7fc640 10 monclient: dump: 2024-03-28T16:15:44.877 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-28T16:15:44.877 INFO:teuthology.misc.health.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:44.877 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:44.877 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:44.877 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:44.877 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:44.877 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:44.877 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:44.877 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:44.877 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-28T16:15:44.877 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0bbfff640 4 set_mon_vals no callback set 2024-03-28T16:15:44.877 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0cc0d4640 10 monclient: _renew_subs 2024-03-28T16:15:44.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0cc0d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:44.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0cc0d4640 1 -- 172.21.15.7:0/1747986334 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c4143390 con 0x7ff0c4141e50 2024-03-28T16:15:44.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0cc0d4640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:44.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0cc0d4640 10 monclient: _renew_subs 2024-03-28T16:15:44.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0cc0d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:44.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0cc0d4640 1 -- 172.21.15.7:0/1747986334 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c40b0d30 con 0x7ff0c4141e50 2024-03-28T16:15:44.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.873+0000 7ff0cc0d4640 1 librados: init done 2024-03-28T16:15:44.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.874+0000 7ff0cc0d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:44.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.874+0000 7ff0cc0d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:44.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.874+0000 7ff0cc0d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:44.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.874+0000 7ff0cc0d4640 1 -- 172.21.15.7:0/1747986334 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c4140880 con 0x7ff0c4141e50 2024-03-28T16:15:44.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.891+0000 7ff0ba7fc640 1 -- 172.21.15.7:0/1747986334 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff0bc03b050 con 0x7ff0c4141e50 2024-03-28T16:15:44.892 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.892+0000 7ff0ba7fc640 1 --2- 172.21.15.7:0/1747986334 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7ff0a806c5e0 0x7ff0a806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:44.892 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.892+0000 7ff0c9e70640 1 --2- 172.21.15.7:0/1747986334 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7ff0a806c5e0 0x7ff0a806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:44.892 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.892+0000 7ff0c9e70640 10 monclient: get_auth_request con 0x7ff0a806c5e0 auth_method 0 2024-03-28T16:15:44.892 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.892+0000 7ff0c9e70640 1 --2- 172.21.15.7:0/1747986334 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7ff0a806c5e0 0x7ff0a806eaa0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff0b40017f0 tx=0x7ff0b4009040 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:44.893 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:44.892+0000 7ff0ba7fc640 10 client.4279.objecter ms_handle_connect 0x7ff0a806c5e0 2024-03-28T16:15:45.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.125+0000 7ff0ba7fc640 1 -- 172.21.15.7:0/1747986334 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff0bc07ac20 con 0x7ff0c4141e50 2024-03-28T16:15:45.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.125+0000 7ff0ba7fc640 10 client.4279.objecter ms_dispatch 0x7ff0c4000b90 osd_map(25..25 src has 1..25) v4 2024-03-28T16:15:45.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.125+0000 7ff0ba7fc640 3 client.4279.objecter handle_osd_map got epochs [25,25] > 0 2024-03-28T16:15:45.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.125+0000 7ff0ba7fc640 3 client.4279.objecter handle_osd_map decoding full epoch 25 2024-03-28T16:15:45.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.125+0000 7ff0ba7fc640 20 client.4279.objecter dump_active .. 0 homeless 2024-03-28T16:15:45.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.137+0000 7ff0ba7fc640 1 -- 172.21.15.7:0/1747986334 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff0bc076f60 con 0x7ff0c4141e50 2024-03-28T16:15:45.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.137+0000 7ff0ba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:45.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.137+0000 7ff0ba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:45.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.323+0000 7ff0cc0d4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:45.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.323+0000 7ff0cc0d4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:45.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.323+0000 7ff0cc0d4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:45.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.323+0000 7ff0cc0d4640 1 -- 172.21.15.7:0/1747986334 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0c4140880 con 0x7ff0c4141e50 2024-03-28T16:15:45.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.350+0000 7ff0ba7fc640 1 -- 172.21.15.7:0/1747986334 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+12286 (secure 0 0 0) 0x7ff0c4140880 con 0x7ff0c4141e50 2024-03-28T16:15:45.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.350+0000 7ff0ba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:45.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.350+0000 7ff0ba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:45.350 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-28T16:15:45.350 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":25,"fsid":"808ff1a1-2b17-4c35-a89a-340dbbf674ee","created":"2024-03-28T16:14:44.591599+0000","modified":"2024-03-28T16:15:43.313152+0000","last_up_change":"2024-03-28T16:15:40.088013+0000","last_in_change":"2024-03-28T16:14:49.757700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T16:15:31.373047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a19c6856-78b9-4919-97ce-6940a5af982c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":34918},{"type":"v1","addr":"172.21.15.7:6809","nonce":34918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":34918},{"type":"v1","addr":"172.21.15.7:6811","nonce":34918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":34918},{"type":"v1","addr":"172.21.15.7:6815","nonce":34918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":34918},{"type":"v1","addr":"172.21.15.7:6813","nonce":34918}]},"public_addr":"172.21.15.7:6809/34918","cluster_addr":"172.21.15.7:6811/34918","heartbeat_back_addr":"172.21.15.7:6815/34918","heartbeat_front_addr":"172.21.15.7:6813/34918","state":["exists","up"]},{"osd":1,"uuid":"2afee73d-3369-4fd2-bfc5-36dee610daf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":34919},{"type":"v1","addr":"172.21.15.7:6801","nonce":34919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":34919},{"type":"v1","addr":"172.21.15.7:6803","nonce":34919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":34919},{"type":"v1","addr":"172.21.15.7:6807","nonce":34919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":34919},{"type":"v1","addr":"172.21.15.7:6805","nonce":34919}]},"public_addr":"172.21.15.7:6801/34919","cluster_addr":"172.21.15.7:6803/34919","heartbeat_back_addr":"172.21.15.7:6807/34919","heartbeat_front_addr":"172.21.15.7:6805/34919","state":["exists","up"]},{"osd":2,"uuid":"0473d84c-e864-4373-b9d1-283130278363","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":34920},{"type":"v1","addr":"172.21.15.7:6825","nonce":34920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":34920},{"type":"v1","addr":"172.21.15.7:6827","nonce":34920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":34920},{"type":"v1","addr":"172.21.15.7:6831","nonce":34920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":34920},{"type":"v1","addr":"172.21.15.7:6829","nonce":34920}]},"public_addr":"172.21.15.7:6825/34920","cluster_addr":"172.21.15.7:6827/34920","heartbeat_back_addr":"172.21.15.7:6831/34920","heartbeat_front_addr":"172.21.15.7:6829/34920","state":["exists","up"]},{"osd":3,"uuid":"b64fd5e7-cdec-4bd0-b5e2-dfc8d4f6b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":34921},{"type":"v1","addr":"172.21.15.7:6817","nonce":34921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":34921},{"type":"v1","addr":"172.21.15.7:6819","nonce":34921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":34921},{"type":"v1","addr":"172.21.15.7:6823","nonce":34921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":34921},{"type":"v1","addr":"172.21.15.7:6821","nonce":34921}]},"public_addr":"172.21.15.7:6817/34921","cluster_addr":"172.21.15.7:6819/34921","heartbeat_back_addr":"172.21.15.7:6823/34921","heartbeat_front_addr":"172.21.15.7:6821/34921","state":["exists","up"]},{"osd":4,"uuid":"8f7552db-5651-430f-a826-3c6843bffb99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":35294},{"type":"v1","addr":"172.21.15.83:6819","nonce":35294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":35294},{"type":"v1","addr":"172.21.15.83:6821","nonce":35294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":35294},{"type":"v1","addr":"172.21.15.83:6825","nonce":35294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":35294},{"type":"v1","addr":"172.21.15.83:6823","nonce":35294}]},"public_addr":"172.21.15.83:6819/35294","cluster_addr":"172.21.15.83:6821/35294","heartbeat_back_addr":"172.21.15.83:6825/35294","heartbeat_front_addr":"172.21.15.83:6823/35294","state":["exists","up"]},{"osd":5,"uuid":"2aabdbd2-d870-4d6d-823e-89b008ea3108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":35296},{"type":"v1","addr":"172.21.15.83:6827","nonce":35296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":35296},{"type":"v1","addr":"172.21.15.83:6829","nonce":35296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":35296},{"type":"v1","addr":"172.21.15.83:6833","nonce":35296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":35296},{"type":"v1","addr":"172.21.15.83:6831","nonce":35296}]},"public_addr":"172.21.15.83:6827/35296","cluster_addr":"172.21.15.83:6829/35296","heartbeat_back_addr":"172.21.15.83:6833/35296","heartbeat_front_addr":"172.21.15.83:6831/35296","state":["exists","up"]},{"osd":6,"uuid":"fbbb7557-08d4-48d4-9b88-6bb1d680a87d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":35297},{"type":"v1","addr":"172.21.15.83:6811","nonce":35297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":35297},{"type":"v1","addr":"172.21.15.83:6813","nonce":35297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":35297},{"type":"v1","addr":"172.21.15.83:6817","nonce":35297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":35297},{"type":"v1","addr":"172.21.15.83:6815","nonce":35297}]},"public_addr":"172.21.15.83:6811/35297","cluster_addr":"172.21.15.83:6813/35297","heartbeat_back_addr":"172.21.15.83:6817/35297","heartbeat_front_addr":"172.21.15.83:6815/35297","state":["exists","up"]},{"osd":7,"uuid":"805f54d9-03d5-4b23-aef3-e1d55bb09f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":35295},{"type":"v1","addr":"172.21.15.83:6803","nonce":35295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":35295},{"type":"v1","addr":"172.21.15.83:6805","nonce":35295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":35295},{"type":"v1","addr":"172.21.15.83:6809","nonce":35295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":35295},{"type":"v1","addr":"172.21.15.83:6807","nonce":35295}]},"public_addr":"172.21.15.83:6803/35295","cluster_addr":"172.21.15.83:6805/35295","heartbeat_back_addr":"172.21.15.83:6809/35295","heartbeat_front_addr":"172.21.15.83:6807/35295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:27.198013+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:28.569762+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:28.085669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:28.165751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:26.831000+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:28.235232+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:27.056704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:26.424892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:15:45.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.351+0000 7ff0cc0d4640 10 librados: watch_flush enter 2024-03-28T16:15:45.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.352+0000 7ff0cc0d4640 10 librados: watch_flush exit 2024-03-28T16:15:45.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.352+0000 7ff0cc0d4640 20 client.4279.objecter shutdown clearing up homeless session... 2024-03-28T16:15:45.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.352+0000 7ff0cc0d4640 10 client.4279.objecter successfully canceled tick 2024-03-28T16:15:45.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.352+0000 7ff0cc0d4640 1 -- 172.21.15.7:0/1747986334 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7ff0a806c5e0 msgr2=0x7ff0a806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:45.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.352+0000 7ff0cc0d4640 1 --2- 172.21.15.7:0/1747986334 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7ff0a806c5e0 0x7ff0a806eaa0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff0b40017f0 tx=0x7ff0b4009040 comp rx=0 tx=0).stop 2024-03-28T16:15:45.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.352+0000 7ff0cc0d4640 10 monclient: shutdown 2024-03-28T16:15:45.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.352+0000 7ff0cc0d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:45.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.352+0000 7ff0cc0d4640 1 -- 172.21.15.7:0/1747986334 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0c4141e50 msgr2=0x7ff0c4143fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:45.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.352+0000 7ff0cc0d4640 1 --2- 172.21.15.7:0/1747986334 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0c4141e50 0x7ff0c4143fd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc009e80 tx=0x7ff0bc004a40 comp rx=0 tx=0).stop 2024-03-28T16:15:45.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.352+0000 7ff0cc0d4640 1 -- 172.21.15.7:0/1747986334 shutdown_connections 2024-03-28T16:15:45.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.352+0000 7ff0cc0d4640 1 --2- 172.21.15.7:0/1747986334 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7ff0a806c5e0 0x7ff0a806eaa0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:45.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.352+0000 7ff0cc0d4640 1 --2- 172.21.15.7:0/1747986334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff0c413eeb0 0x7ff0c4143a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:45.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.352+0000 7ff0cc0d4640 1 --2- 172.21.15.7:0/1747986334 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0c4141e50 0x7ff0c4143fd0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:45.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.352+0000 7ff0cc0d4640 1 --2- 172.21.15.7:0/1747986334 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff0c40719a0 0x7ff0c4140170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:45.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.352+0000 7ff0cc0d4640 1 -- 172.21.15.7:0/1747986334 >> 172.21.15.7:0/1747986334 conn(0x7ff0c40b72b0 msgr2=0x7ff0c4138df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:45.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.352+0000 7ff0cc0d4640 1 -- 172.21.15.7:0/1747986334 shutdown_connections 2024-03-28T16:15:45.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.352+0000 7ff0cc0d4640 1 -- 172.21.15.7:0/1747986334 wait complete. 2024-03-28T16:15:45.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-28T16:15:45.352+0000 7ff0cc0d4640 1 librados: shutdown 2024-03-28T16:15:45.361 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-28T16:15:45.361 INFO:tasks.ceph:Creating RBD pool 2024-03-28T16:15:45.362 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd pool create rbd 8 2024-03-28T16:15:45.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.474+0000 7f4270f34640 10 monclient: get_monmap_and_config 2024-03-28T16:15:45.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.474+0000 7f4270f34640 10 monclient: build_initial_monmap 2024-03-28T16:15:45.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.474+0000 7f4270f34640 10 monclient: monmap: 2024-03-28T16:15:45.474 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:45.474 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:45.475 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:45.475082+0000 2024-03-28T16:15:45.475 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:45.475082+0000 2024-03-28T16:15:45.475 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:45.475 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:45.475 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:45.475 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:45.475 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:45.475 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:45.475 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:45.475 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:45.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.475+0000 7f4270f34640 1 Processor -- start 2024-03-28T16:15:45.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.475+0000 7f4270f34640 1 -- start start 2024-03-28T16:15:45.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.475+0000 7f4270f34640 10 monclient: init 2024-03-28T16:15:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.475+0000 7f4270f34640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.475+0000 7f4270f34640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-03-28T16:15:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.475+0000 7f4270f34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f426c071480 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.475+0000 7f4270f34640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f426c0b32e0 0x7f426c144a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.475+0000 7f4270f34640 10 monclient(hunting): picked mon.noname-a con 0x7f426c0b32e0 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.475+0000 7f4270f34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f426c0a86e0 addr v1:172.21.15.83:6789/0 2024-03-28T16:15:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.475+0000 7f4270f34640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f426c0a8c90 con 0x7f426c071480 2024-03-28T16:15:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.475+0000 7f4270f34640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f426c145650 con 0x7f426c0a86e0 2024-03-28T16:15:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.475+0000 7f4270f34640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.475+0000 7f4270f34640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f426c071860 con 0x7f426c0b32e0 2024-03-28T16:15:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.475+0000 7f4270f34640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.475+0000 7f4270f34640 10 monclient(hunting): authenticate will time out at 1412.616713s 2024-03-28T16:15:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.477+0000 7f4269d9b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f426c0b32e0 0x7f426c144a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:45.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.477+0000 7f4269d9b640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f426c0b32e0 0x7f426c144a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58110/0 (socket says 172.21.15.7:58110) 2024-03-28T16:15:45.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.477+0000 7f4269d9b640 1 -- 172.21.15.7:0/3073693465 learned_addr learned my addr 172.21.15.7:0/3073693465 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:45.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.477+0000 7f4269d9b640 10 monclient(hunting): get_auth_request con 0x7f426c0b32e0 auth_method 0 2024-03-28T16:15:45.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.477+0000 7f4269d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:45.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.477+0000 7f4269d9b640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:45.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.477+0000 7f4269d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:45.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.478+0000 7f4269d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:45.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.478+0000 7f4269d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:45.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.478+0000 7f4269d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:45.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.480+0000 7f4269d9b640 10 monclient(hunting): handle_auth_done global_id 4281 payload 274 2024-03-28T16:15:45.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.480+0000 7f4269d9b640 1 -- 172.21.15.7:0/3073693465 >> v1:172.21.15.83:6789/0 conn(0x7f426c0a86e0 legacy=0x7f426c144f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:45.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.480+0000 7f4269d9b640 1 -- 172.21.15.7:0/3073693465 >> v1:172.21.15.7:6789/0 conn(0x7f426c071480 legacy=0x7f426c13e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:45.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.480+0000 7f4269d9b640 10 monclient: _finish_hunting 0 2024-03-28T16:15:45.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.480+0000 7f4269d9b640 1 monclient: found mon.noname-a 2024-03-28T16:15:45.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.480+0000 7f4269d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:45.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.480+0000 7f4269d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:15:45.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.480+0000 7f4269d9b640 1 -- 172.21.15.7:0/3073693465 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f426c13f020 con 0x7f426c0b32e0 2024-03-28T16:15:45.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.486+0000 7f4269d9b640 1 --2- 172.21.15.7:0/3073693465 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f426c0b32e0 0x7f426c144a00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4260009870 tx=0x7f426002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abf34ebfb5a37f6c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:45.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.493+0000 7f426959a640 1 -- 172.21.15.7:0/3073693465 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4260031870 con 0x7f426c0b32e0 2024-03-28T16:15:45.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.493+0000 7f426959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:45.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.493+0000 7f426959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:15:45.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.493+0000 7f426959a640 10 monclient: dump: 2024-03-28T16:15:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:45.495 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.493+0000 7f426959a640 10 monclient: _finish_auth 0 2024-03-28T16:15:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.493+0000 7f426959a640 10 monclient: _check_auth_tickets 2024-03-28T16:15:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.493+0000 7f426959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.493+0000 7f426959a640 1 -- 172.21.15.7:0/3073693465 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42600319d0 con 0x7f426c0b32e0 2024-03-28T16:15:45.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.493+0000 7f4270f34640 5 monclient: authenticate success, global_id 4281 2024-03-28T16:15:45.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.493+0000 7f4270f34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:45.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.493+0000 7f426959a640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:45.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.493+0000 7f4270f34640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:45.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.493+0000 7f4270f34640 4 set_mon_vals no callback set 2024-03-28T16:15:45.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.493+0000 7f426959a640 1 -- 172.21.15.7:0/3073693465 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4260031d60 con 0x7f426c0b32e0 2024-03-28T16:15:45.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.493+0000 7f426959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:45.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.493+0000 7f426959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:15:45.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.493+0000 7f426959a640 10 monclient: dump: 2024-03-28T16:15:45.496 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:45.496 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:45.496 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:45.496 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:45.496 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:45.496 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:45.496 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:45.497 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.494+0000 7f4270f34640 10 monclient: shutdown 2024-03-28T16:15:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.494+0000 7f4270f34640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.494+0000 7f4270f34640 1 -- 172.21.15.7:0/3073693465 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f426c0b32e0 msgr2=0x7f426c144a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.494+0000 7f4270f34640 1 --2- 172.21.15.7:0/3073693465 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f426c0b32e0 0x7f426c144a00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4260009870 tx=0x7f426002b910 comp rx=0 tx=0).stop 2024-03-28T16:15:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.494+0000 7f4270f34640 1 -- 172.21.15.7:0/3073693465 shutdown_connections 2024-03-28T16:15:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.494+0000 7f4270f34640 1 --2- 172.21.15.7:0/3073693465 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f426c0b32e0 0x7f426c144a00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.494+0000 7f4270f34640 1 -- 172.21.15.7:0/3073693465 >> 172.21.15.7:0/3073693465 conn(0x7f426c0b70e0 msgr2=0x7f426c0b74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.494+0000 7f4270f34640 1 -- 172.21.15.7:0/3073693465 shutdown_connections 2024-03-28T16:15:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.494+0000 7f4270f34640 1 -- 172.21.15.7:0/3073693465 wait complete. 2024-03-28T16:15:45.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.495+0000 7f4270f34640 10 monclient: build_initial_monmap 2024-03-28T16:15:45.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.495+0000 7f4270f34640 10 monclient: monmap: 2024-03-28T16:15:45.499 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:45.499 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:45.499 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:45.499 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:45.499 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:45.499 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:45.499 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:45.500 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:45.500 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:45.500 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:45.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.495+0000 7f4270f34640 1 librados: starting msgr at 2024-03-28T16:15:45.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.495+0000 7f4270f34640 1 librados: starting objecter 2024-03-28T16:15:45.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.495+0000 7f4270f34640 1 Processor -- start 2024-03-28T16:15:45.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.495+0000 7f4270f34640 1 -- start start 2024-03-28T16:15:45.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.495+0000 7f4270f34640 1 librados: setting wanted keys 2024-03-28T16:15:45.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.495+0000 7f4270f34640 1 librados: calling monclient init 2024-03-28T16:15:45.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.495+0000 7f4270f34640 10 monclient: init 2024-03-28T16:15:45.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.496+0000 7f4270f34640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:45.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.496+0000 7f4270f34640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-28T16:15:45.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.496+0000 7f4270f34640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f426c071480 0x7f426c134a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:45.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.496+0000 7f4270f34640 10 monclient(hunting): picked mon.noname-b con 0x7f426c071480 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:45.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.496+0000 7f4270f34640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f426c0a86e0 0x7f426c135550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:45.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.496+0000 7f4270f34640 10 monclient(hunting): picked mon.noname-a con 0x7f426c0a86e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:45.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.496+0000 7f4270f34640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f426c0b32e0 0x7f426c1443a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:45.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.496+0000 7f4270f34640 10 monclient(hunting): picked mon.noname-c con 0x7f426c0b32e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:45.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.496+0000 7f4270f34640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:45.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.496+0000 7f4270f34640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c13fd00 con 0x7f426c0a86e0 2024-03-28T16:15:45.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.496+0000 7f4270f34640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:45.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.496+0000 7f4270f34640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c13ffe0 con 0x7f426c071480 2024-03-28T16:15:45.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.496+0000 7f4270f34640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:45.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.496+0000 7f4270f34640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f426c1402c0 con 0x7f426c0b32e0 2024-03-28T16:15:45.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.496+0000 7f4270f34640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:45.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.496+0000 7f4270f34640 10 monclient(hunting): authenticate will time out at 1412.637217s 2024-03-28T16:15:45.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.496+0000 7f426ad9d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f426c0b32e0 0x7f426c1443a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:45.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.497+0000 7f426ad9d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f426c0b32e0 0x7f426c1443a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:39142/0 (socket says 172.21.15.7:39142) 2024-03-28T16:15:45.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.497+0000 7f426ad9d640 1 -- 172.21.15.7:0/1992160029 learned_addr learned my addr 172.21.15.7:0/1992160029 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:45.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.497+0000 7f426ad9d640 10 monclient(hunting): get_auth_request con 0x7f426c0b32e0 auth_method 0 2024-03-28T16:15:45.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.497+0000 7f426ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:45.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.497+0000 7f426ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:45.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.497+0000 7f426ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:45.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.497+0000 7f426ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:45.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.497+0000 7f426ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:45.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.497+0000 7f426ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:45.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.500+0000 7f426ad9d640 10 monclient(hunting): handle_auth_done global_id 4247 payload 824 2024-03-28T16:15:45.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.500+0000 7f426ad9d640 1 -- 172.21.15.7:0/1992160029 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f426c071480 msgr2=0x7f426c134a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:45.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.500+0000 7f426ad9d640 1 --2- 172.21.15.7:0/1992160029 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f426c071480 0x7f426c134a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:45.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.500+0000 7f426ad9d640 1 -- 172.21.15.7:0/1992160029 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f426c0a86e0 msgr2=0x7f426c135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:45.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.500+0000 7f426ad9d640 1 --2- 172.21.15.7:0/1992160029 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f426c0a86e0 0x7f426c135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:45.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.500+0000 7f426ad9d640 10 monclient: _finish_hunting 0 2024-03-28T16:15:45.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.500+0000 7f426ad9d640 1 monclient: found mon.noname-c 2024-03-28T16:15:45.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.500+0000 7f426ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:45.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.500+0000 7f426ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-03-28T16:15:45.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.500+0000 7f426ad9d640 1 -- 172.21.15.7:0/1992160029 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42600094d0 con 0x7f426c0b32e0 2024-03-28T16:15:45.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.502+0000 7f426ad9d640 1 --2- 172.21.15.7:0/1992160029 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f426c0b32e0 0x7f426c1443a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f426400c7a0 tx=0x7f426400cc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:45.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.502+0000 7f425affd640 10 client.?.objecter ms_handle_connect 0x7f426c0b32e0 2024-03-28T16:15:45.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.502+0000 7f425affd640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:45.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f425affd640 1 -- 172.21.15.7:0/1992160029 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4264003b70 con 0x7f426c0b32e0 2024-03-28T16:15:45.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f425affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:45.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f425affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:45.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f425affd640 10 monclient: dump: 2024-03-28T16:15:45.507 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:45.507 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:45.507 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:45.507 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:45.507 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:45.507 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:45.507 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:45.507 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:45.507 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:45.507 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:45.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f425affd640 10 monclient: _finish_auth 0 2024-03-28T16:15:45.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f425affd640 10 monclient: _check_auth_tickets 2024-03-28T16:15:45.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f425affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:45.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f425affd640 1 -- 172.21.15.7:0/1992160029 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4264003cf0 con 0x7f426c0b32e0 2024-03-28T16:15:45.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f425affd640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:45.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f425affd640 1 -- 172.21.15.7:0/1992160029 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4264003ea0 con 0x7f426c0b32e0 2024-03-28T16:15:45.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f4270f34640 5 monclient: authenticate success, global_id 4247 2024-03-28T16:15:45.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f4270f34640 10 monclient: _renew_subs 2024-03-28T16:15:45.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f4270f34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:45.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f4270f34640 1 -- 172.21.15.7:0/1992160029 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f426c1429c0 con 0x7f426c0b32e0 2024-03-28T16:15:45.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f425affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:45.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f425affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:15:45.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f4268d99640 4 set_mon_vals no callback set 2024-03-28T16:15:45.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f425affd640 10 monclient: dump: 2024-03-28T16:15:45.508 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:45.508 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:45.508 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:45.508 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:45.509 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:45.509 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:45.509 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:45.509 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:45.509 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:45.509 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:45.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f4270f34640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:45.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f4270f34640 10 monclient: _renew_subs 2024-03-28T16:15:45.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f4270f34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:45.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f4270f34640 1 -- 172.21.15.7:0/1992160029 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f426c143570 con 0x7f426c0b32e0 2024-03-28T16:15:45.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.506+0000 7f4270f34640 1 librados: init done 2024-03-28T16:15:45.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.507+0000 7f4270f34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:45.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.507+0000 7f4270f34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:45.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.507+0000 7f4270f34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:45.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.507+0000 7f4270f34640 1 -- 172.21.15.7:0/1992160029 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f426c139ef0 con 0x7f426c0b32e0 2024-03-28T16:15:45.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.526+0000 7f425affd640 1 -- 172.21.15.7:0/1992160029 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f426400aa90 con 0x7f426c0b32e0 2024-03-28T16:15:45.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.526+0000 7f425affd640 1 --2- 172.21.15.7:0/1992160029 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f424c06c5e0 0x7f424c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:45.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.526+0000 7f426a59c640 1 --2- 172.21.15.7:0/1992160029 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f424c06c5e0 0x7f424c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:45.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.526+0000 7f426a59c640 10 monclient: get_auth_request con 0x7f424c06c5e0 auth_method 0 2024-03-28T16:15:45.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.527+0000 7f426a59c640 1 --2- 172.21.15.7:0/1992160029 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f424c06c5e0 0x7f424c06eaa0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f425c000f60 tx=0x7f425c0079b0 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:45.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.527+0000 7f425affd640 10 client.4247.objecter ms_handle_connect 0x7f424c06c5e0 2024-03-28T16:15:45.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.529+0000 7f425affd640 1 -- 172.21.15.7:0/1992160029 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4264059220 con 0x7f426c0b32e0 2024-03-28T16:15:45.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.529+0000 7f425affd640 10 client.4247.objecter ms_dispatch 0x7f426c000b90 osd_map(25..25 src has 1..25) v4 2024-03-28T16:15:45.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.529+0000 7f425affd640 3 client.4247.objecter handle_osd_map got epochs [25,25] > 0 2024-03-28T16:15:45.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.529+0000 7f425affd640 3 client.4247.objecter handle_osd_map decoding full epoch 25 2024-03-28T16:15:45.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.529+0000 7f425affd640 20 client.4247.objecter dump_active .. 0 homeless 2024-03-28T16:15:45.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.787+0000 7f425affd640 1 -- 172.21.15.7:0/1992160029 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4264054e60 con 0x7f426c0b32e0 2024-03-28T16:15:45.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.788+0000 7f425affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:45.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.788+0000 7f425affd640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:45.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.969+0000 7f4270f34640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-03-28T16:15:45.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.969+0000 7f4270f34640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-03-28T16:15:45.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.969+0000 7f4270f34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:45.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:45.969+0000 7f4270f34640 1 -- 172.21.15.7:0/1992160029 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f426c143ff0 con 0x7f426c0b32e0 2024-03-28T16:15:46.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.613+0000 7f425affd640 1 -- 172.21.15.7:0/1992160029 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v26) v1 ==== 109+0+0 (secure 0 0 0) 0x7f4264058480 con 0x7f426c0b32e0 2024-03-28T16:15:46.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.613+0000 7f425affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-03-28T16:15:46.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.613+0000 7f425affd640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-03-28T16:15:46.613 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'rbd' created 2024-03-28T16:15:46.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.615+0000 7f4270f34640 10 librados: watch_flush enter 2024-03-28T16:15:46.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.615+0000 7f4270f34640 10 librados: watch_flush exit 2024-03-28T16:15:46.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.615+0000 7f4270f34640 20 client.4247.objecter shutdown clearing up homeless session... 2024-03-28T16:15:46.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.615+0000 7f4270f34640 10 client.4247.objecter successfully canceled tick 2024-03-28T16:15:46.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.615+0000 7f4270f34640 1 -- 172.21.15.7:0/1992160029 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f424c06c5e0 msgr2=0x7f424c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:46.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.615+0000 7f4270f34640 1 --2- 172.21.15.7:0/1992160029 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f424c06c5e0 0x7f424c06eaa0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f425c000f60 tx=0x7f425c0079b0 comp rx=0 tx=0).stop 2024-03-28T16:15:46.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.615+0000 7f4270f34640 10 monclient: shutdown 2024-03-28T16:15:46.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.615+0000 7f4270f34640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:46.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.615+0000 7f4270f34640 1 -- 172.21.15.7:0/1992160029 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f426c0b32e0 msgr2=0x7f426c1443a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:46.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.615+0000 7f4270f34640 1 --2- 172.21.15.7:0/1992160029 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f426c0b32e0 0x7f426c1443a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f426400c7a0 tx=0x7f426400cc60 comp rx=0 tx=0).stop 2024-03-28T16:15:46.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.615+0000 7f4270f34640 1 -- 172.21.15.7:0/1992160029 shutdown_connections 2024-03-28T16:15:46.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.615+0000 7f4270f34640 1 --2- 172.21.15.7:0/1992160029 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f424c06c5e0 0x7f424c06eaa0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:46.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.615+0000 7f4270f34640 1 --2- 172.21.15.7:0/1992160029 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f426c0a86e0 0x7f426c135550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:46.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.615+0000 7f4270f34640 1 --2- 172.21.15.7:0/1992160029 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f426c0b32e0 0x7f426c1443a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:46.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.615+0000 7f4270f34640 1 --2- 172.21.15.7:0/1992160029 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f426c071480 0x7f426c134a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:46.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.615+0000 7f4270f34640 1 -- 172.21.15.7:0/1992160029 >> 172.21.15.7:0/1992160029 conn(0x7f426c0b70e0 msgr2=0x7f426c0b4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:46.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.615+0000 7f4270f34640 1 -- 172.21.15.7:0/1992160029 shutdown_connections 2024-03-28T16:15:46.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.616+0000 7f4270f34640 1 -- 172.21.15.7:0/1992160029 wait complete. 2024-03-28T16:15:46.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.616+0000 7f4270f34640 1 librados: shutdown 2024-03-28T16:15:46.627 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-28T16:15:46.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.729+0000 7f733d021640 10 monclient: get_monmap_and_config 2024-03-28T16:15:46.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.729+0000 7f733d021640 10 monclient: build_initial_monmap 2024-03-28T16:15:46.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.729+0000 7f733d021640 10 monclient: monmap: 2024-03-28T16:15:46.730 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:46.730 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:46.730 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:46.730065+0000 2024-03-28T16:15:46.730 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:46.730065+0000 2024-03-28T16:15:46.730 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:46.730 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:46.730 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:46.730 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:46.730 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:46.730 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:46.730 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:46.730 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:46.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.730+0000 7f733d021640 1 Processor -- start 2024-03-28T16:15:46.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.730+0000 7f733d021640 1 -- start start 2024-03-28T16:15:46.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.730+0000 7f733d021640 10 monclient: init 2024-03-28T16:15:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.730+0000 7f733d021640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.730+0000 7f733d021640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-03-28T16:15:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.730+0000 7f733d021640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f733806a5f0 0x7f733806b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.730+0000 7f733d021640 10 monclient(hunting): picked mon.noname-c con 0x7f733806a5f0 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.730+0000 7f733d021640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f733806a9d0 0x7f73380ad9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.730+0000 7f733d021640 10 monclient(hunting): picked mon.noname-a con 0x7f733806a9d0 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.730+0000 7f733d021640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7338141b90 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:46.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.730+0000 7f733d021640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73380af2e0 con 0x7f7338141b90 2024-03-28T16:15:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.730+0000 7f733d021640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.730+0000 7f733d021640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73380ad600 con 0x7f733806a9d0 2024-03-28T16:15:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.730+0000 7f733d021640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.730+0000 7f733d021640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f733806b390 con 0x7f733806a5f0 2024-03-28T16:15:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.730+0000 7f733d021640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.730+0000 7f733d021640 10 monclient(hunting): authenticate will time out at 1413.871579s 2024-03-28T16:15:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.731+0000 7f7336d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f733806a5f0 0x7f733806b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.731+0000 7f733659c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f733806a9d0 0x7f73380ad9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.731+0000 7f7336d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f733806a5f0 0x7f733806b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:57896/0 (socket says 172.21.15.7:57896) 2024-03-28T16:15:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.731+0000 7f7336d9d640 1 -- 172.21.15.7:0/1976508561 learned_addr learned my addr 172.21.15.7:0/1976508561 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.731+0000 7f7336d9d640 10 monclient(hunting): get_auth_request con 0x7f733806a5f0 auth_method 0 2024-03-28T16:15:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.731+0000 7f7336d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.731+0000 7f7336d9d640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.731+0000 7f7336d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:46.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.732+0000 7f7336d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.732+0000 7f7336d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.732+0000 7f7336d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.732+0000 7f733659c640 10 monclient(hunting): get_auth_request con 0x7f733806a9d0 auth_method 0 2024-03-28T16:15:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.732+0000 7f733659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.732+0000 7f733659c640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.732+0000 7f733659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.732+0000 7f733659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.732+0000 7f733659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.732+0000 7f733659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.733+0000 7f7336d9d640 10 monclient(hunting): handle_auth_done global_id 4282 payload 274 2024-03-28T16:15:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.733+0000 7f7336d9d640 1 -- 172.21.15.7:0/1976508561 >> v2:172.21.15.7:3300/0 conn(0x7f733806a9d0 msgr2=0x7f73380ad9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.733+0000 7f7336d9d640 1 --2- 172.21.15.7:0/1976508561 >> v2:172.21.15.7:3300/0 conn(0x7f733806a9d0 0x7f73380ad9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.733+0000 7f7336d9d640 1 -- 172.21.15.7:0/1976508561 >> v1:172.21.15.7:6789/0 conn(0x7f7338141b90 legacy=0x7f733813e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.733+0000 7f7336d9d640 10 monclient: _finish_hunting 0 2024-03-28T16:15:46.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.733+0000 7f7336d9d640 1 monclient: found mon.noname-c 2024-03-28T16:15:46.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.733+0000 7f7336d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:46.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.733+0000 7f7336d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3300/0 2024-03-28T16:15:46.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.733+0000 7f7336d9d640 1 -- 172.21.15.7:0/1976508561 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733813ee50 con 0x7f733806a5f0 2024-03-28T16:15:46.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.736+0000 7f7336d9d640 1 --2- 172.21.15.7:0/1976508561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733806a5f0 0x7f733806b7e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f73380b5c10 tx=0x7f732802b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56bc6cd9fe2c6495 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:46.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.739+0000 7f7335d9b640 1 -- 172.21.15.7:0/1976508561 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73280317e0 con 0x7f733806a5f0 2024-03-28T16:15:46.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.739+0000 7f7335d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:46.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.739+0000 7f7335d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:46.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.739+0000 7f7335d9b640 10 monclient: dump: 2024-03-28T16:15:46.740 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:46.740 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:46.740 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:46.741 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.739+0000 7f7335d9b640 10 monclient: _finish_auth 0 2024-03-28T16:15:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.739+0000 7f7335d9b640 10 monclient: _check_auth_tickets 2024-03-28T16:15:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.739+0000 7f7335d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.739+0000 7f7335d9b640 1 -- 172.21.15.7:0/1976508561 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7328031940 con 0x7f733806a5f0 2024-03-28T16:15:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.739+0000 7f7335d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:46.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.739+0000 7f7335d9b640 1 -- 172.21.15.7:0/1976508561 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7328031cd0 con 0x7f733806a5f0 2024-03-28T16:15:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.739+0000 7f733d021640 5 monclient: authenticate success, global_id 4282 2024-03-28T16:15:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.739+0000 7f733d021640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.739+0000 7f733d021640 4 set_mon_vals no callback set 2024-03-28T16:15:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.739+0000 7f7335d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.739+0000 7f7335d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.739+0000 7f7335d9b640 10 monclient: dump: 2024-03-28T16:15:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:46.742 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:46.743 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.740+0000 7f733d021640 10 monclient: shutdown 2024-03-28T16:15:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.740+0000 7f733d021640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.740+0000 7f733d021640 1 -- 172.21.15.7:0/1976508561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733806a5f0 msgr2=0x7f733806b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.740+0000 7f733d021640 1 --2- 172.21.15.7:0/1976508561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733806a5f0 0x7f733806b7e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f73380b5c10 tx=0x7f732802b880 comp rx=0 tx=0).stop 2024-03-28T16:15:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.740+0000 7f733d021640 1 -- 172.21.15.7:0/1976508561 shutdown_connections 2024-03-28T16:15:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.740+0000 7f733d021640 1 --2- 172.21.15.7:0/1976508561 >> v2:172.21.15.7:3300/0 conn(0x7f733806a9d0 0x7f73380ad9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.740+0000 7f733d021640 1 --2- 172.21.15.7:0/1976508561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733806a5f0 0x7f733806b7e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.740+0000 7f733d021640 1 -- 172.21.15.7:0/1976508561 >> 172.21.15.7:0/1976508561 conn(0x7f73380b5080 msgr2=0x7f73380b5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.740+0000 7f733d021640 1 -- 172.21.15.7:0/1976508561 shutdown_connections 2024-03-28T16:15:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.740+0000 7f733d021640 1 -- 172.21.15.7:0/1976508561 wait complete. 2024-03-28T16:15:46.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.741+0000 7f733d021640 10 monclient: build_initial_monmap 2024-03-28T16:15:46.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.741+0000 7f733d021640 10 monclient: monmap: 2024-03-28T16:15:46.747 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:46.747 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:46.747 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:46.747 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:46.747 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:46.747 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:46.747 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:46.747 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:46.747 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:46.747 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:46.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.741+0000 7f733d021640 1 librados: starting msgr at 2024-03-28T16:15:46.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.741+0000 7f733d021640 1 librados: starting objecter 2024-03-28T16:15:46.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.741+0000 7f733d021640 1 Processor -- start 2024-03-28T16:15:46.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.741+0000 7f733d021640 1 -- start start 2024-03-28T16:15:46.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.741+0000 7f733d021640 1 librados: setting wanted keys 2024-03-28T16:15:46.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.741+0000 7f733d021640 1 librados: calling monclient init 2024-03-28T16:15:46.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.741+0000 7f733d021640 10 monclient: init 2024-03-28T16:15:46.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.742+0000 7f733d021640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:46.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.742+0000 7f733d021640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-28T16:15:46.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.742+0000 7f733d021640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733806a5f0 0x7f733813a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:46.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.742+0000 7f733d021640 10 monclient(hunting): picked mon.noname-b con 0x7f733806a5f0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:46.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.742+0000 7f733d021640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f733806a9d0 0x7f7338136280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:46.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.742+0000 7f733d021640 10 monclient(hunting): picked mon.noname-a con 0x7f733806a9d0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:46.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.742+0000 7f733d021640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7338141b90 0x7f73381367c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:46.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.742+0000 7f733d021640 10 monclient(hunting): picked mon.noname-c con 0x7f7338141b90 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:46.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.742+0000 7f733d021640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:46.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.742+0000 7f733d021640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7338137c00 con 0x7f733806a9d0 2024-03-28T16:15:46.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.742+0000 7f733d021640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:46.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.742+0000 7f733d021640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7338137ee0 con 0x7f733806a5f0 2024-03-28T16:15:46.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.742+0000 7f733d021640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:46.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.742+0000 7f733d021640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f733813a780 con 0x7f7338141b90 2024-03-28T16:15:46.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.742+0000 7f733d021640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:46.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.742+0000 7f733d021640 10 monclient(hunting): authenticate will time out at 1413.883254s 2024-03-28T16:15:46.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.742+0000 7f733759e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7338141b90 0x7f73381367c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:46.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.742+0000 7f7336d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733806a5f0 0x7f733813a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:46.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.743+0000 7f733659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f733806a9d0 0x7f7338136280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:46.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.743+0000 7f7336d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733806a5f0 0x7f733813a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:57908/0 (socket says 172.21.15.7:57908) 2024-03-28T16:15:46.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.743+0000 7f7336d9d640 1 -- 172.21.15.7:0/1549552591 learned_addr learned my addr 172.21.15.7:0/1549552591 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:46.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.743+0000 7f7336d9d640 10 monclient(hunting): get_auth_request con 0x7f733806a5f0 auth_method 0 2024-03-28T16:15:46.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.743+0000 7f7336d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:46.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.743+0000 7f7336d9d640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:46.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.743+0000 7f7336d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:46.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.743+0000 7f733759e640 10 monclient(hunting): get_auth_request con 0x7f7338141b90 auth_method 0 2024-03-28T16:15:46.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.743+0000 7f733759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:46.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.743+0000 7f733759e640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:46.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.743+0000 7f733759e640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:46.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.743+0000 7f733659c640 10 monclient(hunting): get_auth_request con 0x7f733806a9d0 auth_method 0 2024-03-28T16:15:46.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.743+0000 7f733659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:46.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.743+0000 7f733659c640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:46.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.743+0000 7f733659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:46.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.743+0000 7f7336d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:46.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.743+0000 7f7336d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:46.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.743+0000 7f7336d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:46.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.743+0000 7f733759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:46.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.743+0000 7f733759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:46.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.743+0000 7f733759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:46.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.744+0000 7f733659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:46.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.744+0000 7f733659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:46.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.744+0000 7f733659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:46.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.746+0000 7f7336d9d640 10 monclient(hunting): handle_auth_done global_id 4285 payload 824 2024-03-28T16:15:46.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.746+0000 7f7336d9d640 1 -- 172.21.15.7:0/1549552591 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7338141b90 msgr2=0x7f73381367c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:46.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.746+0000 7f7336d9d640 1 --2- 172.21.15.7:0/1549552591 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7338141b90 0x7f73381367c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:46.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.746+0000 7f7336d9d640 1 -- 172.21.15.7:0/1549552591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f733806a9d0 msgr2=0x7f7338136280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:46.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.746+0000 7f7336d9d640 1 --2- 172.21.15.7:0/1549552591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f733806a9d0 0x7f7338136280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:46.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.746+0000 7f7336d9d640 10 monclient: _finish_hunting 0 2024-03-28T16:15:46.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.746+0000 7f7336d9d640 1 monclient: found mon.noname-b 2024-03-28T16:15:46.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.746+0000 7f7336d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:46.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.746+0000 7f7336d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:15:46.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.746+0000 7f7336d9d640 1 -- 172.21.15.7:0/1549552591 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73280094d0 con 0x7f733806a5f0 2024-03-28T16:15:46.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.749+0000 7f7336d9d640 1 --2- 172.21.15.7:0/1549552591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733806a5f0 0x7f733813a120 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f732802b8c0 tx=0x7f73280046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:46.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.749+0000 7f731f7fe640 10 client.?.objecter ms_handle_connect 0x7f733806a5f0 2024-03-28T16:15:46.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.749+0000 7f731f7fe640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:46.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f731f7fe640 1 -- 172.21.15.7:0/1549552591 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7328032880 con 0x7f733806a5f0 2024-03-28T16:15:46.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f731f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:46.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f731f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:46.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f731f7fe640 10 monclient: dump: 2024-03-28T16:15:46.754 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:46.754 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:46.754 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:46.754 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:46.754 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:46.754 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:46.754 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:46.754 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:46.754 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:46.754 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:46.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f731f7fe640 10 monclient: _finish_auth 0 2024-03-28T16:15:46.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f731f7fe640 10 monclient: _check_auth_tickets 2024-03-28T16:15:46.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f731f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:46.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f731f7fe640 1 -- 172.21.15.7:0/1549552591 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7328032a20 con 0x7f733806a5f0 2024-03-28T16:15:46.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f731f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:46.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f731f7fe640 1 -- 172.21.15.7:0/1549552591 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7328032dd0 con 0x7f733806a5f0 2024-03-28T16:15:46.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f733d021640 5 monclient: authenticate success, global_id 4285 2024-03-28T16:15:46.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f733d021640 10 monclient: _renew_subs 2024-03-28T16:15:46.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f733d021640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:46.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f733d021640 1 -- 172.21.15.7:0/1549552591 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733806f010 con 0x7f733806a5f0 2024-03-28T16:15:46.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f731f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:46.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f731f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:46.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f733559a640 4 set_mon_vals no callback set 2024-03-28T16:15:46.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f731f7fe640 10 monclient: dump: 2024-03-28T16:15:46.755 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:46.755 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:46.756 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:46.756 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:46.756 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:46.756 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:46.756 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:46.756 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:46.756 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:46.756 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:46.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f733d021640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:46.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f733d021640 10 monclient: _renew_subs 2024-03-28T16:15:46.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f733d021640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:46.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f733d021640 1 -- 172.21.15.7:0/1549552591 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7338136ee0 con 0x7f733806a5f0 2024-03-28T16:15:46.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.752+0000 7f733d021640 1 librados: init done 2024-03-28T16:15:46.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.753+0000 7f733d021640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:46.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.753+0000 7f733d021640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:46.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.753+0000 7f733d021640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:46.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.753+0000 7f733d021640 1 -- 172.21.15.7:0/1549552591 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73380acda0 con 0x7f733806a5f0 2024-03-28T16:15:46.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.772+0000 7f731f7fe640 1 -- 172.21.15.7:0/1549552591 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f732804e450 con 0x7f733806a5f0 2024-03-28T16:15:46.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.772+0000 7f731f7fe640 1 --2- 172.21.15.7:0/1549552591 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f732406c590 0x7f732406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:46.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.772+0000 7f733659c640 1 --2- 172.21.15.7:0/1549552591 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f732406c590 0x7f732406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:46.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.772+0000 7f733659c640 10 monclient: get_auth_request con 0x7f732406c590 auth_method 0 2024-03-28T16:15:46.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.773+0000 7f733659c640 1 --2- 172.21.15.7:0/1549552591 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f732406c590 0x7f732406ea50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f733813ba80 tx=0x7f732c009190 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:46.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:46.773+0000 7f731f7fe640 10 client.4285.objecter ms_handle_connect 0x7f732406c590 2024-03-28T16:15:47.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:47.005+0000 7f731f7fe640 1 -- 172.21.15.7:0/1549552591 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f732807a510 con 0x7f733806a5f0 2024-03-28T16:15:47.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:47.005+0000 7f731f7fe640 10 client.4285.objecter ms_dispatch 0x7f7338000b90 osd_map(26..26 src has 1..26) v4 2024-03-28T16:15:47.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:47.005+0000 7f731f7fe640 3 client.4285.objecter handle_osd_map got epochs [26,26] > 0 2024-03-28T16:15:47.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:47.005+0000 7f731f7fe640 3 client.4285.objecter handle_osd_map decoding full epoch 26 2024-03-28T16:15:47.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:47.005+0000 7f731f7fe640 20 client.4285.objecter dump_active .. 0 homeless 2024-03-28T16:15:47.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:47.017+0000 7f731f7fe640 1 -- 172.21.15.7:0/1549552591 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7328076850 con 0x7f733806a5f0 2024-03-28T16:15:47.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:47.017+0000 7f731f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:47.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:47.017+0000 7f731f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:47.167 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:19.739 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:15:47.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:47.219+0000 7f733d021640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-03-28T16:15:47.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:47.219+0000 7f733d021640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-03-28T16:15:47.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:47.219+0000 7f733d021640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:47.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:47.219+0000 7f733d021640 1 -- 172.21.15.7:0/1549552591 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7338136ee0 con 0x7f733806a5f0 2024-03-28T16:15:47.226 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:19.799 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:15:47.260 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:19.833 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:15:47.268 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:01:19.824 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:15:47.318 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:19.891 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:15:47.335 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:01:19.891 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:15:47.572 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:20.145 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:15:47.599 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:20.172 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:15:47.602 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:01:20.159 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:15:48.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.238+0000 7f731f7fe640 1 -- 172.21.15.7:0/1549552591 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v27) v1 ==== 173+0+0 (secure 0 0 0) 0x7f7338136ee0 con 0x7f733806a5f0 2024-03-28T16:15:48.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.238+0000 7f731f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-03-28T16:15:48.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.238+0000 7f731f7fe640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-03-28T16:15:48.239 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-28T16:15:48.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.240+0000 7f733d021640 10 librados: watch_flush enter 2024-03-28T16:15:48.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.240+0000 7f733d021640 10 librados: watch_flush exit 2024-03-28T16:15:48.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.240+0000 7f733d021640 20 client.4285.objecter shutdown clearing up homeless session... 2024-03-28T16:15:48.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.240+0000 7f733d021640 10 client.4285.objecter successfully canceled tick 2024-03-28T16:15:48.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.240+0000 7f733d021640 1 -- 172.21.15.7:0/1549552591 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f732406c590 msgr2=0x7f732406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:48.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.240+0000 7f733d021640 1 --2- 172.21.15.7:0/1549552591 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f732406c590 0x7f732406ea50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f733813ba80 tx=0x7f732c009190 comp rx=0 tx=0).stop 2024-03-28T16:15:48.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.240+0000 7f733d021640 10 monclient: shutdown 2024-03-28T16:15:48.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.240+0000 7f733d021640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:48.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.240+0000 7f733d021640 1 -- 172.21.15.7:0/1549552591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733806a5f0 msgr2=0x7f733813a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:48.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.240+0000 7f733d021640 1 --2- 172.21.15.7:0/1549552591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733806a5f0 0x7f733813a120 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f732802b8c0 tx=0x7f73280046c0 comp rx=0 tx=0).stop 2024-03-28T16:15:48.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.241+0000 7f733d021640 1 -- 172.21.15.7:0/1549552591 shutdown_connections 2024-03-28T16:15:48.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.241+0000 7f733d021640 1 --2- 172.21.15.7:0/1549552591 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f732406c590 0x7f732406ea50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:48.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.241+0000 7f733d021640 1 --2- 172.21.15.7:0/1549552591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f733806a9d0 0x7f7338136280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:48.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.241+0000 7f733d021640 1 --2- 172.21.15.7:0/1549552591 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7338141b90 0x7f73381367c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:48.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.241+0000 7f733d021640 1 --2- 172.21.15.7:0/1549552591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f733806a5f0 0x7f733813a120 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:48.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.241+0000 7f733d021640 1 -- 172.21.15.7:0/1549552591 >> 172.21.15.7:0/1549552591 conn(0x7f73380b5080 msgr2=0x7f73380ae820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:48.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.241+0000 7f733d021640 1 -- 172.21.15.7:0/1549552591 shutdown_connections 2024-03-28T16:15:48.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.241+0000 7f733d021640 1 -- 172.21.15.7:0/1549552591 wait complete. 2024-03-28T16:15:48.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.241+0000 7f733d021640 1 librados: shutdown 2024-03-28T16:15:48.254 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-28T16:15:48.254 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-28T16:15:48.254 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-28T16:15:48.255 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-28T16:15:48.255 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-28T16:15:48.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.356+0000 7f3800be4640 10 monclient: get_monmap_and_config 2024-03-28T16:15:48.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.356+0000 7f3800be4640 10 monclient: build_initial_monmap 2024-03-28T16:15:48.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.356+0000 7f3800be4640 10 monclient: monmap: 2024-03-28T16:15:48.358 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:48.358 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:48.358 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:48.358081+0000 2024-03-28T16:15:48.358 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:48.358081+0000 2024-03-28T16:15:48.358 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:48.358 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:48.358 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:48.358 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:48.358 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:48.359 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:48.359 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:48.359 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:48.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.357+0000 7f3800be4640 1 Processor -- start 2024-03-28T16:15:48.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.357+0000 7f3800be4640 1 -- start start 2024-03-28T16:15:48.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.357+0000 7f3800be4640 10 monclient: init 2024-03-28T16:15:48.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.357+0000 7f3800be4640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:48.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.357+0000 7f3800be4640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-03-28T16:15:48.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.357+0000 7f3800be4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f37fc071480 0x7f37fc13e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:48.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.357+0000 7f3800be4640 10 monclient(hunting): picked mon.noname-a con 0x7f37fc071480 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:48.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.357+0000 7f3800be4640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37fc0b32e0 0x7f37fc144a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:48.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.357+0000 7f3800be4640 10 monclient(hunting): picked mon.noname-b con 0x7f37fc0b32e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:48.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.357+0000 7f3800be4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f37fc13ee50 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:48.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.357+0000 7f3800be4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37fc0a8c90 con 0x7f37fc13ee50 2024-03-28T16:15:48.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.357+0000 7f3800be4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:48.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.357+0000 7f3800be4640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37fc071860 con 0x7f37fc071480 2024-03-28T16:15:48.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.357+0000 7f3800be4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:48.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.357+0000 7f3800be4640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc145650 con 0x7f37fc0b32e0 2024-03-28T16:15:48.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.357+0000 7f3800be4640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:48.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.357+0000 7f3800be4640 10 monclient(hunting): authenticate will time out at 1415.499804s 2024-03-28T16:15:48.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.358+0000 7f37fa59c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f37fc071480 0x7f37fc13e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:48.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.359+0000 7f37fa59c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f37fc071480 0x7f37fc13e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37758/0 (socket says 172.21.15.7:37758) 2024-03-28T16:15:48.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.359+0000 7f37fa59c640 1 -- 172.21.15.7:0/2757470552 learned_addr learned my addr 172.21.15.7:0/2757470552 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:48.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.359+0000 7f37fa59c640 10 monclient(hunting): get_auth_request con 0x7f37fc071480 auth_method 0 2024-03-28T16:15:48.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.359+0000 7f37fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:48.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.359+0000 7f37fa59c640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:48.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.359+0000 7f37fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:48.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.360+0000 7f37fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:48.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.360+0000 7f37fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:48.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.360+0000 7f37fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:48.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.363+0000 7f37fa59c640 10 monclient(hunting): handle_auth_done global_id 4293 payload 274 2024-03-28T16:15:48.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.363+0000 7f37fa59c640 1 -- 172.21.15.7:0/2757470552 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37fc0b32e0 msgr2=0x7f37fc144a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:48.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.363+0000 7f37fa59c640 1 --2- 172.21.15.7:0/2757470552 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37fc0b32e0 0x7f37fc144a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:48.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.363+0000 7f37fa59c640 1 -- 172.21.15.7:0/2757470552 >> v1:172.21.15.7:6789/0 conn(0x7f37fc13ee50 legacy=0x7f37fc144f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:48.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.363+0000 7f37fa59c640 10 monclient: _finish_hunting 0 2024-03-28T16:15:48.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.363+0000 7f37fa59c640 1 monclient: found mon.noname-a 2024-03-28T16:15:48.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.363+0000 7f37fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:48.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.363+0000 7f37fa59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:15:48.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.363+0000 7f37fa59c640 1 -- 172.21.15.7:0/2757470552 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37fc072140 con 0x7f37fc071480 2024-03-28T16:15:48.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.368+0000 7f37fa59c640 1 --2- 172.21.15.7:0/2757470552 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37fc071480 0x7f37fc13e910 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f37ec009870 tx=0x7f37ec02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f62419539f68580 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:48.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.373+0000 7f37f959a640 1 -- 172.21.15.7:0/2757470552 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec0317c0 con 0x7f37fc071480 2024-03-28T16:15:48.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.373+0000 7f37f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:48.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.373+0000 7f37f959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:15:48.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.373+0000 7f37f959a640 10 monclient: dump: 2024-03-28T16:15:48.375 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:48.375 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:48.375 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:48.375 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:48.378 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:48.378 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:48.378 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:48.378 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:48.378 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:48.378 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:48.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.373+0000 7f37f959a640 10 monclient: _finish_auth 0 2024-03-28T16:15:48.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.373+0000 7f37f959a640 10 monclient: _check_auth_tickets 2024-03-28T16:15:48.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.373+0000 7f37f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:48.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.373+0000 7f37f959a640 1 -- 172.21.15.7:0/2757470552 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ec031920 con 0x7f37fc071480 2024-03-28T16:15:48.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.373+0000 7f37f959a640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:48.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.373+0000 7f37f959a640 1 -- 172.21.15.7:0/2757470552 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec031cb0 con 0x7f37fc071480 2024-03-28T16:15:48.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.373+0000 7f37f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:48.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.373+0000 7f37f959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:15:48.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.373+0000 7f37f959a640 10 monclient: dump: 2024-03-28T16:15:48.379 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:48.379 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:48.379 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:48.379 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:48.379 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:48.379 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:48.379 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:48.379 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:48.379 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:48.380 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:48.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.373+0000 7f3800be4640 5 monclient: authenticate success, global_id 4293 2024-03-28T16:15:48.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.373+0000 7f3800be4640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:48.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.373+0000 7f3800be4640 4 set_mon_vals no callback set 2024-03-28T16:15:48.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.374+0000 7f3800be4640 10 monclient: shutdown 2024-03-28T16:15:48.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.374+0000 7f3800be4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:48.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.374+0000 7f3800be4640 1 -- 172.21.15.7:0/2757470552 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37fc071480 msgr2=0x7f37fc13e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:48.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.374+0000 7f3800be4640 1 --2- 172.21.15.7:0/2757470552 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37fc071480 0x7f37fc13e910 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f37ec009870 tx=0x7f37ec02b860 comp rx=0 tx=0).stop 2024-03-28T16:15:48.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.374+0000 7f3800be4640 1 -- 172.21.15.7:0/2757470552 shutdown_connections 2024-03-28T16:15:48.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.374+0000 7f3800be4640 1 --2- 172.21.15.7:0/2757470552 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37fc0b32e0 0x7f37fc144a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:48.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.374+0000 7f3800be4640 1 --2- 172.21.15.7:0/2757470552 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37fc071480 0x7f37fc13e910 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:48.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.374+0000 7f3800be4640 1 -- 172.21.15.7:0/2757470552 >> 172.21.15.7:0/2757470552 conn(0x7f37fc0b70e0 msgr2=0x7f37fc0b74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:48.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.374+0000 7f3800be4640 1 -- 172.21.15.7:0/2757470552 shutdown_connections 2024-03-28T16:15:48.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.374+0000 7f3800be4640 1 -- 172.21.15.7:0/2757470552 wait complete. 2024-03-28T16:15:48.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.375+0000 7f3800be4640 10 monclient: build_initial_monmap 2024-03-28T16:15:48.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.375+0000 7f3800be4640 10 monclient: monmap: 2024-03-28T16:15:48.385 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:48.385 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:48.385 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:48.386 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:48.386 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:48.386 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:48.386 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:48.386 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:48.386 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:48.386 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:48.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.375+0000 7f3800be4640 1 librados: starting msgr at 2024-03-28T16:15:48.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.375+0000 7f3800be4640 1 librados: starting objecter 2024-03-28T16:15:48.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.375+0000 7f3800be4640 1 Processor -- start 2024-03-28T16:15:48.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.376+0000 7f3800be4640 1 -- start start 2024-03-28T16:15:48.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.376+0000 7f3800be4640 1 librados: setting wanted keys 2024-03-28T16:15:48.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.376+0000 7f3800be4640 1 librados: calling monclient init 2024-03-28T16:15:48.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.376+0000 7f3800be4640 10 monclient: init 2024-03-28T16:15:48.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.376+0000 7f3800be4640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:48.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.376+0000 7f3800be4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-28T16:15:48.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.376+0000 7f3800be4640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37fc071480 0x7f37fc143530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:48.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.376+0000 7f3800be4640 10 monclient(hunting): picked mon.noname-c con 0x7f37fc071480 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:48.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.376+0000 7f3800be4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37fc0b32e0 0x7f37fc143c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:48.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.376+0000 7f3800be4640 10 monclient(hunting): picked mon.noname-a con 0x7f37fc0b32e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:48.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.376+0000 7f3800be4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37fc13ee50 0x7f37fc13dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:48.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.376+0000 7f3800be4640 10 monclient(hunting): picked mon.noname-b con 0x7f37fc13ee50 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:48.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.376+0000 7f3800be4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:48.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.376+0000 7f3800be4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc134e40 con 0x7f37fc0b32e0 2024-03-28T16:15:48.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.376+0000 7f3800be4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:48.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.376+0000 7f3800be4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc13d930 con 0x7f37fc13ee50 2024-03-28T16:15:48.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.376+0000 7f3800be4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:48.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.376+0000 7f3800be4640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc13c410 con 0x7f37fc071480 2024-03-28T16:15:48.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.376+0000 7f3800be4640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:48.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.376+0000 7f3800be4640 10 monclient(hunting): authenticate will time out at 1415.518400s 2024-03-28T16:15:48.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.377+0000 7f37fa59c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37fc071480 0x7f37fc143530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:48.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.377+0000 7f37f9d9b640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37fc0b32e0 0x7f37fc143c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:48.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.377+0000 7f37fa59c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37fc071480 0x7f37fc143530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60052/0 (socket says 172.21.15.7:60052) 2024-03-28T16:15:48.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.377+0000 7f37fa59c640 1 -- 172.21.15.7:0/3752723341 learned_addr learned my addr 172.21.15.7:0/3752723341 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:48.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.377+0000 7f37fa59c640 10 monclient(hunting): get_auth_request con 0x7f37fc071480 auth_method 0 2024-03-28T16:15:48.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.377+0000 7f37fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:48.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.377+0000 7f37fa59c640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:48.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.377+0000 7f37fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:48.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.377+0000 7f37fad9d640 1 --2- 172.21.15.7:0/3752723341 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37fc13ee50 0x7f37fc13dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:48.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.377+0000 7f37f9d9b640 10 monclient(hunting): get_auth_request con 0x7f37fc0b32e0 auth_method 0 2024-03-28T16:15:48.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.377+0000 7f37f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:48.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.377+0000 7f37f9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:48.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.377+0000 7f37f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:48.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.377+0000 7f37fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:48.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.377+0000 7f37fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:48.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.377+0000 7f37fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:48.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.378+0000 7f37f9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:48.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.378+0000 7f37f9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:48.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.378+0000 7f37f9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:48.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.378+0000 7f37fad9d640 10 monclient(hunting): get_auth_request con 0x7f37fc13ee50 auth_method 0 2024-03-28T16:15:48.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.378+0000 7f37fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:48.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.378+0000 7f37fad9d640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:48.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.378+0000 7f37fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:48.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.378+0000 7f37fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:48.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.378+0000 7f37fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:48.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.378+0000 7f37fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:48.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.380+0000 7f37fa59c640 10 monclient(hunting): handle_auth_done global_id 4253 payload 824 2024-03-28T16:15:48.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.380+0000 7f37fa59c640 1 -- 172.21.15.7:0/3752723341 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37fc13ee50 msgr2=0x7f37fc13dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:48.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.380+0000 7f37fa59c640 1 --2- 172.21.15.7:0/3752723341 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37fc13ee50 0x7f37fc13dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:48.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.380+0000 7f37fa59c640 1 -- 172.21.15.7:0/3752723341 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37fc0b32e0 msgr2=0x7f37fc143c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:48.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.380+0000 7f37fa59c640 1 --2- 172.21.15.7:0/3752723341 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37fc0b32e0 0x7f37fc143c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:48.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.380+0000 7f37fa59c640 10 monclient: _finish_hunting 0 2024-03-28T16:15:48.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.380+0000 7f37fa59c640 1 monclient: found mon.noname-c 2024-03-28T16:15:48.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.380+0000 7f37fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:48.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.381+0000 7f37fa59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-03-28T16:15:48.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.381+0000 7f37fa59c640 1 -- 172.21.15.7:0/3752723341 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ec0094d0 con 0x7f37fc071480 2024-03-28T16:15:48.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.383+0000 7f37fa59c640 1 --2- 172.21.15.7:0/3752723341 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37fc071480 0x7f37fc143530 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f37ec02b8a0 tx=0x7f37ec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:48.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.383+0000 7f37eaffd640 10 client.?.objecter ms_handle_connect 0x7f37fc071480 2024-03-28T16:15:48.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.383+0000 7f37eaffd640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:48.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.388+0000 7f37eaffd640 1 -- 172.21.15.7:0/3752723341 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec032960 con 0x7f37fc071480 2024-03-28T16:15:48.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.388+0000 7f37eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:48.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.388+0000 7f37eaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:48.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.388+0000 7f37eaffd640 10 monclient: dump: 2024-03-28T16:15:48.393 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:48.393 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:48.393 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:48.393 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:48.393 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:48.393 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:48.393 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:48.393 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:48.393 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:48.393 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:48.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.388+0000 7f37eaffd640 10 monclient: _finish_auth 0 2024-03-28T16:15:48.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.388+0000 7f37eaffd640 10 monclient: _check_auth_tickets 2024-03-28T16:15:48.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.388+0000 7f37eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:48.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.388+0000 7f37eaffd640 1 -- 172.21.15.7:0/3752723341 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ec032ae0 con 0x7f37fc071480 2024-03-28T16:15:48.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.388+0000 7f3800be4640 5 monclient: authenticate success, global_id 4253 2024-03-28T16:15:48.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.388+0000 7f3800be4640 10 monclient: _renew_subs 2024-03-28T16:15:48.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.388+0000 7f3800be4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:48.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.388+0000 7f3800be4640 1 -- 172.21.15.7:0/3752723341 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37fc13e2a0 con 0x7f37fc071480 2024-03-28T16:15:48.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.388+0000 7f37eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:48.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.388+0000 7f37f8d99640 4 set_mon_vals no callback set 2024-03-28T16:15:48.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.388+0000 7f3800be4640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:48.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.388+0000 7f3800be4640 10 monclient: _renew_subs 2024-03-28T16:15:48.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.388+0000 7f3800be4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:48.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.388+0000 7f3800be4640 1 -- 172.21.15.7:0/3752723341 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37fc1429e0 con 0x7f37fc071480 2024-03-28T16:15:48.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.388+0000 7f3800be4640 1 librados: init done 2024-03-28T16:15:48.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.388+0000 7f37eaffd640 1 -- 172.21.15.7:0/3752723341 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec032e90 con 0x7f37fc071480 2024-03-28T16:15:48.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.389+0000 7f37eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:48.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.389+0000 7f37eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:15:48.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.389+0000 7f37eaffd640 10 monclient: dump: 2024-03-28T16:15:48.395 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:48.395 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:48.395 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:48.395 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:48.395 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:48.395 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:48.395 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:48.395 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:48.395 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:48.395 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:48.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.389+0000 7f3800be4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:48.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.389+0000 7f3800be4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:48.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.389+0000 7f3800be4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:48.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.389+0000 7f3800be4640 1 -- 172.21.15.7:0/3752723341 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37fc04e0c0 con 0x7f37fc071480 2024-03-28T16:15:48.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.670+0000 7f37eaffd640 1 -- 172.21.15.7:0/3752723341 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f37ec04d3f0 con 0x7f37fc071480 2024-03-28T16:15:48.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.670+0000 7f37eaffd640 1 --2- 172.21.15.7:0/3752723341 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f37dc06c5e0 0x7f37dc06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:48.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.671+0000 7f37f9d9b640 1 --2- 172.21.15.7:0/3752723341 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f37dc06c5e0 0x7f37dc06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:48.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.671+0000 7f37f9d9b640 10 monclient: get_auth_request con 0x7f37dc06c5e0 auth_method 0 2024-03-28T16:15:48.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.671+0000 7f37f9d9b640 1 --2- 172.21.15.7:0/3752723341 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f37dc06c5e0 0x7f37dc06eaa0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f37fc13d440 tx=0x7f37f0009110 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:48.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.671+0000 7f37eaffd640 10 client.4253.objecter ms_handle_connect 0x7f37dc06c5e0 2024-03-28T16:15:48.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.683+0000 7f37eaffd640 1 -- 172.21.15.7:0/3752723341 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f37ec07abd0 con 0x7f37fc071480 2024-03-28T16:15:48.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.683+0000 7f37eaffd640 10 client.4253.objecter ms_dispatch 0x7f37fc000b90 osd_map(27..27 src has 1..27) v4 2024-03-28T16:15:48.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.683+0000 7f37eaffd640 3 client.4253.objecter handle_osd_map got epochs [27,27] > 0 2024-03-28T16:15:48.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.683+0000 7f37eaffd640 3 client.4253.objecter handle_osd_map decoding full epoch 27 2024-03-28T16:15:48.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.683+0000 7f37eaffd640 20 client.4253.objecter dump_active .. 0 homeless 2024-03-28T16:15:48.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.683+0000 7f37eaffd640 1 -- 172.21.15.7:0/3752723341 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f37ec076810 con 0x7f37fc071480 2024-03-28T16:15:48.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.683+0000 7f37eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:48.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.683+0000 7f37eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:48.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.957+0000 7f3800be4640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-03-28T16:15:48.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.957+0000 7f3800be4640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-03-28T16:15:48.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.958+0000 7f3800be4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:48.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:48.958+0000 7f3800be4640 1 -- 172.21.15.7:0/3752723341 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f37fc067040 con 0x7f37fc071480 2024-03-28T16:15:49.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.125+0000 7f37eaffd640 1 -- 172.21.15.7:0/3752723341 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v7) v1 ==== 74+0+170926 (secure 0 0 0) 0x7f37ec079e30 con 0x7f37fc071480 2024-03-28T16:15:49.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.125+0000 7f37eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-03-28T16:15:49.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.125+0000 7f37eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:49.129 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:15:49.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.128+0000 7f3800be4640 10 librados: watch_flush enter 2024-03-28T16:15:49.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.128+0000 7f3800be4640 10 librados: watch_flush exit 2024-03-28T16:15:49.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.128+0000 7f3800be4640 20 client.4253.objecter shutdown clearing up homeless session... 2024-03-28T16:15:49.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.128+0000 7f3800be4640 10 client.4253.objecter successfully canceled tick 2024-03-28T16:15:49.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.128+0000 7f3800be4640 1 -- 172.21.15.7:0/3752723341 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f37dc06c5e0 msgr2=0x7f37dc06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:49.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.128+0000 7f3800be4640 1 --2- 172.21.15.7:0/3752723341 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f37dc06c5e0 0x7f37dc06eaa0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f37fc13d440 tx=0x7f37f0009110 comp rx=0 tx=0).stop 2024-03-28T16:15:49.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.128+0000 7f3800be4640 10 monclient: shutdown 2024-03-28T16:15:49.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.128+0000 7f3800be4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:49.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.128+0000 7f3800be4640 1 -- 172.21.15.7:0/3752723341 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37fc071480 msgr2=0x7f37fc143530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:49.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.128+0000 7f3800be4640 1 --2- 172.21.15.7:0/3752723341 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37fc071480 0x7f37fc143530 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f37ec02b8a0 tx=0x7f37ec0046c0 comp rx=0 tx=0).stop 2024-03-28T16:15:49.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.128+0000 7f3800be4640 1 -- 172.21.15.7:0/3752723341 shutdown_connections 2024-03-28T16:15:49.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.128+0000 7f3800be4640 1 --2- 172.21.15.7:0/3752723341 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f37dc06c5e0 0x7f37dc06eaa0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:49.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.128+0000 7f3800be4640 1 --2- 172.21.15.7:0/3752723341 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37fc0b32e0 0x7f37fc143c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:49.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.128+0000 7f3800be4640 1 --2- 172.21.15.7:0/3752723341 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37fc13ee50 0x7f37fc13dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:49.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.128+0000 7f3800be4640 1 --2- 172.21.15.7:0/3752723341 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37fc071480 0x7f37fc143530 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:49.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.128+0000 7f3800be4640 1 -- 172.21.15.7:0/3752723341 >> 172.21.15.7:0/3752723341 conn(0x7f37fc0b70e0 msgr2=0x7f37fc0b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:49.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.128+0000 7f3800be4640 1 -- 172.21.15.7:0/3752723341 shutdown_connections 2024-03-28T16:15:49.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.128+0000 7f3800be4640 1 -- 172.21.15.7:0/3752723341 wait complete. 2024-03-28T16:15:49.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.128+0000 7f3800be4640 1 librados: shutdown 2024-03-28T16:15:49.140 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":7,"active_gid":4162,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":34752},{"type":"v1","addr":"172.21.15.83:6801","nonce":34752}]},"active_addr":"172.21.15.83:6801/34752","active_change":"2024-03-28T16:14:56.782295+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4112,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":11,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":2497381276}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":1340604352}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":78143101}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":2892510762}]}]} 2024-03-28T16:15:49.143 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-28T16:15:49.143 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-28T16:15:49.143 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:15:49.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.251+0000 7f703415c640 10 monclient: get_monmap_and_config 2024-03-28T16:15:49.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.251+0000 7f703415c640 10 monclient: build_initial_monmap 2024-03-28T16:15:49.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.251+0000 7f703415c640 10 monclient: monmap: 2024-03-28T16:15:49.253 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:49.253 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:49.253 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:49.253479+0000 2024-03-28T16:15:49.253 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:49.253479+0000 2024-03-28T16:15:49.253 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:49.253 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:49.253 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:49.253 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:49.253 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:49.253 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:49.253 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:49.253 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:49.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.252+0000 7f703415c640 1 Processor -- start 2024-03-28T16:15:49.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.252+0000 7f703415c640 1 -- start start 2024-03-28T16:15:49.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.252+0000 7f703415c640 10 monclient: init 2024-03-28T16:15:49.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.253+0000 7f703415c640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:49.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.253+0000 7f703415c640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-03-28T16:15:49.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.253+0000 7f703415c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f702c071480 addr v1:172.21.15.83:6789/0 2024-03-28T16:15:49.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.253+0000 7f703415c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f702c0b32e0 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:49.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.253+0000 7f703415c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f702c0a86e0 0x7f702c145110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:49.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.253+0000 7f703415c640 10 monclient(hunting): picked mon.noname-c con 0x7f702c0a86e0 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:49.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.253+0000 7f703415c640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f702c0a8c90 con 0x7f702c0b32e0 2024-03-28T16:15:49.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.253+0000 7f703415c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f702c145650 con 0x7f702c071480 2024-03-28T16:15:49.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.253+0000 7f703415c640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:49.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.253+0000 7f703415c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f702c071860 con 0x7f702c0a86e0 2024-03-28T16:15:49.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.253+0000 7f703415c640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:49.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.253+0000 7f703415c640 10 monclient(hunting): authenticate will time out at 1416.395114s 2024-03-28T16:15:49.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.253+0000 7f70326f9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f702c0a86e0 0x7f702c145110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:49.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.254+0000 7f70326f9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f702c0a86e0 0x7f702c145110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:57916/0 (socket says 172.21.15.7:57916) 2024-03-28T16:15:49.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.254+0000 7f70326f9640 1 -- 172.21.15.7:0/1893354557 learned_addr learned my addr 172.21.15.7:0/1893354557 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:49.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.254+0000 7f70326f9640 10 monclient(hunting): get_auth_request con 0x7f702c0a86e0 auth_method 0 2024-03-28T16:15:49.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.254+0000 7f70326f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:49.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.254+0000 7f70326f9640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:49.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.254+0000 7f70326f9640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:49.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.254+0000 7f70326f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:49.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.254+0000 7f70326f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:49.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.254+0000 7f70326f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:49.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.256+0000 7f70326f9640 10 monclient(hunting): handle_auth_done global_id 4291 payload 274 2024-03-28T16:15:49.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.256+0000 7f70326f9640 1 -- 172.21.15.7:0/1893354557 >> v1:172.21.15.83:6789/0 conn(0x7f702c071480 legacy=0x7f702c13e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:49.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.256+0000 7f70326f9640 1 -- 172.21.15.7:0/1893354557 >> v1:172.21.15.7:6789/0 conn(0x7f702c0b32e0 legacy=0x7f702c144a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:49.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.256+0000 7f70326f9640 10 monclient: _finish_hunting 0 2024-03-28T16:15:49.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.256+0000 7f70326f9640 1 monclient: found mon.noname-c 2024-03-28T16:15:49.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.256+0000 7f70326f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:49.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.256+0000 7f70326f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3300/0 2024-03-28T16:15:49.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.256+0000 7f70326f9640 1 -- 172.21.15.7:0/1893354557 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f702c13f020 con 0x7f702c0a86e0 2024-03-28T16:15:49.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.259+0000 7f70326f9640 1 --2- 172.21.15.7:0/1893354557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f702c0a86e0 0x7f702c145110 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7024009e50 tx=0x7f702402bdc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fcbb6548f30a640 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:49.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.264+0000 7f7030ef6640 1 -- 172.21.15.7:0/1893354557 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702402c960 con 0x7f702c0a86e0 2024-03-28T16:15:49.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.264+0000 7f7030ef6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:49.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.264+0000 7f7030ef6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:49.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.264+0000 7f7030ef6640 10 monclient: dump: 2024-03-28T16:15:49.266 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:49.266 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:49.266 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:49.266 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:49.266 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:49.266 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:49.266 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:49.266 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:49.266 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:49.266 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:49.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.264+0000 7f7030ef6640 10 monclient: _finish_auth 0 2024-03-28T16:15:49.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.264+0000 7f7030ef6640 10 monclient: _check_auth_tickets 2024-03-28T16:15:49.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.264+0000 7f7030ef6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:49.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.264+0000 7f703415c640 5 monclient: authenticate success, global_id 4291 2024-03-28T16:15:49.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.264+0000 7f7030ef6640 1 -- 172.21.15.7:0/1893354557 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f702402cac0 con 0x7f702c0a86e0 2024-03-28T16:15:49.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.264+0000 7f7030ef6640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:49.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.264+0000 7f7030ef6640 1 -- 172.21.15.7:0/1893354557 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702402ce50 con 0x7f702c0a86e0 2024-03-28T16:15:49.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.264+0000 7f703415c640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:49.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.264+0000 7f703415c640 4 set_mon_vals no callback set 2024-03-28T16:15:49.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.264+0000 7f7030ef6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:49.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.264+0000 7f7030ef6640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:49.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.264+0000 7f7030ef6640 10 monclient: dump: 2024-03-28T16:15:49.267 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:49.267 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:49.267 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:49.267 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:49.268 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:49.268 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:49.268 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:49.268 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:49.268 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:49.268 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:49.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.265+0000 7f703415c640 10 monclient: shutdown 2024-03-28T16:15:49.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.265+0000 7f703415c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:49.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.265+0000 7f703415c640 1 -- 172.21.15.7:0/1893354557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f702c0a86e0 msgr2=0x7f702c145110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:49.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.265+0000 7f703415c640 1 --2- 172.21.15.7:0/1893354557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f702c0a86e0 0x7f702c145110 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7024009e50 tx=0x7f702402bdc0 comp rx=0 tx=0).stop 2024-03-28T16:15:49.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.265+0000 7f703415c640 1 -- 172.21.15.7:0/1893354557 shutdown_connections 2024-03-28T16:15:49.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.265+0000 7f703415c640 1 --2- 172.21.15.7:0/1893354557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f702c0a86e0 0x7f702c145110 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:49.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.265+0000 7f703415c640 1 -- 172.21.15.7:0/1893354557 >> 172.21.15.7:0/1893354557 conn(0x7f702c0b70e0 msgr2=0x7f702c0b74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:49.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.265+0000 7f703415c640 1 -- 172.21.15.7:0/1893354557 shutdown_connections 2024-03-28T16:15:49.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.265+0000 7f703415c640 1 -- 172.21.15.7:0/1893354557 wait complete. 2024-03-28T16:15:49.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.266+0000 7f703415c640 10 monclient: build_initial_monmap 2024-03-28T16:15:49.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.266+0000 7f703415c640 10 monclient: monmap: 2024-03-28T16:15:49.272 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:49.272 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:49.272 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:49.272 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:49.272 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:49.272 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:49.272 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:49.272 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:49.273 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:49.273 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:49.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.266+0000 7f703415c640 1 librados: starting msgr at 2024-03-28T16:15:49.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.266+0000 7f703415c640 1 librados: starting objecter 2024-03-28T16:15:49.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.266+0000 7f703415c640 1 Processor -- start 2024-03-28T16:15:49.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.266+0000 7f703415c640 1 -- start start 2024-03-28T16:15:49.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.266+0000 7f703415c640 1 librados: setting wanted keys 2024-03-28T16:15:49.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.266+0000 7f703415c640 1 librados: calling monclient init 2024-03-28T16:15:49.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.266+0000 7f703415c640 10 monclient: init 2024-03-28T16:15:49.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.267+0000 7f703415c640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:49.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.267+0000 7f703415c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-28T16:15:49.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.267+0000 7f703415c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f702c071480 0x7f702c13d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:49.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.267+0000 7f703415c640 10 monclient(hunting): picked mon.noname-a con 0x7f702c071480 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:49.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.267+0000 7f703415c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f702c0a86e0 0x7f702c142c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:49.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.267+0000 7f703415c640 10 monclient(hunting): picked mon.noname-c con 0x7f702c0a86e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:49.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.267+0000 7f703415c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f702c0b32e0 0x7f702c1431c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:49.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.267+0000 7f703415c640 10 monclient(hunting): picked mon.noname-b con 0x7f702c0b32e0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:49.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.267+0000 7f703415c640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:49.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.267+0000 7f703415c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702c13df10 con 0x7f702c071480 2024-03-28T16:15:49.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.267+0000 7f703415c640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:49.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.267+0000 7f703415c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702c143700 con 0x7f702c0b32e0 2024-03-28T16:15:49.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.267+0000 7f703415c640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:49.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.267+0000 7f703415c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f702c13e3a0 con 0x7f702c0a86e0 2024-03-28T16:15:49.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.267+0000 7f703415c640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:49.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.267+0000 7f703415c640 10 monclient(hunting): authenticate will time out at 1416.409266s 2024-03-28T16:15:49.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.267+0000 7f70316f7640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f702c0a86e0 0x7f702c142c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:49.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.267+0000 7f70326f9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f702c0b32e0 0x7f702c1431c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:49.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.268+0000 7f70316f7640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f702c0a86e0 0x7f702c142c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60066/0 (socket says 172.21.15.7:60066) 2024-03-28T16:15:49.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.268+0000 7f70316f7640 1 -- 172.21.15.7:0/2526148944 learned_addr learned my addr 172.21.15.7:0/2526148944 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:49.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.268+0000 7f70316f7640 10 monclient(hunting): get_auth_request con 0x7f702c0a86e0 auth_method 0 2024-03-28T16:15:49.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.268+0000 7f70316f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:49.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.268+0000 7f70316f7640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:49.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.268+0000 7f70316f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:49.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.268+0000 7f7031ef8640 1 --2- 172.21.15.7:0/2526148944 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f702c071480 0x7f702c13d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:49.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.268+0000 7f70326f9640 10 monclient(hunting): get_auth_request con 0x7f702c0b32e0 auth_method 0 2024-03-28T16:15:49.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.268+0000 7f70326f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:49.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.268+0000 7f70326f9640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:49.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.268+0000 7f70326f9640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:49.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.268+0000 7f70316f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:49.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.268+0000 7f70316f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:49.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.268+0000 7f70316f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:49.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.268+0000 7f70326f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:49.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.268+0000 7f70326f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:49.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.268+0000 7f70326f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:49.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.269+0000 7f7031ef8640 10 monclient(hunting): get_auth_request con 0x7f702c071480 auth_method 0 2024-03-28T16:15:49.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.269+0000 7f7031ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:49.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.269+0000 7f7031ef8640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:49.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.269+0000 7f7031ef8640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:49.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.270+0000 7f7031ef8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:49.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.270+0000 7f7031ef8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:49.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.270+0000 7f7031ef8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:49.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.271+0000 7f70326f9640 10 monclient(hunting): handle_auth_done global_id 4294 payload 824 2024-03-28T16:15:49.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.271+0000 7f70326f9640 1 -- 172.21.15.7:0/2526148944 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f702c0a86e0 msgr2=0x7f702c142c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:49.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.271+0000 7f70326f9640 1 --2- 172.21.15.7:0/2526148944 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f702c0a86e0 0x7f702c142c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:49.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.271+0000 7f70326f9640 1 -- 172.21.15.7:0/2526148944 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f702c071480 msgr2=0x7f702c13d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:49.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.271+0000 7f70326f9640 1 --2- 172.21.15.7:0/2526148944 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f702c071480 0x7f702c13d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:49.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.271+0000 7f70326f9640 10 monclient: _finish_hunting 0 2024-03-28T16:15:49.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.271+0000 7f70326f9640 1 monclient: found mon.noname-b 2024-03-28T16:15:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.271+0000 7f70326f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.271+0000 7f70326f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:15:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.271+0000 7f70326f9640 1 -- 172.21.15.7:0/2526148944 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7024009b30 con 0x7f702c0b32e0 2024-03-28T16:15:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.271+0000 7f70316f7640 1 --2- 172.21.15.7:0/2526148944 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f702c0a86e0 0x7f702c142c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:15:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.273+0000 7f70326f9640 1 --2- 172.21.15.7:0/2526148944 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f702c0b32e0 0x7f702c1431c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7024032000 tx=0x7f7024004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.273+0000 7f70227fc640 10 client.?.objecter ms_handle_connect 0x7f702c0b32e0 2024-03-28T16:15:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.273+0000 7f70227fc640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f70227fc640 1 -- 172.21.15.7:0/2526148944 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702403e450 con 0x7f702c0b32e0 2024-03-28T16:15:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f70227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f70227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f70227fc640 10 monclient: dump: 2024-03-28T16:15:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:49.280 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f70227fc640 10 monclient: _finish_auth 0 2024-03-28T16:15:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f70227fc640 10 monclient: _check_auth_tickets 2024-03-28T16:15:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f70227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f70227fc640 1 -- 172.21.15.7:0/2526148944 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f702403e5b0 con 0x7f702c0b32e0 2024-03-28T16:15:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f703415c640 5 monclient: authenticate success, global_id 4294 2024-03-28T16:15:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f703415c640 10 monclient: _renew_subs 2024-03-28T16:15:49.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f703415c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:49.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f703415c640 1 -- 172.21.15.7:0/2526148944 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f702c140120 con 0x7f702c0b32e0 2024-03-28T16:15:49.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f70227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:49.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f70227fc640 1 -- 172.21.15.7:0/2526148944 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702403e980 con 0x7f702c0b32e0 2024-03-28T16:15:49.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f70227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:49.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f70227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:49.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f70227fc640 10 monclient: dump: 2024-03-28T16:15:49.281 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:49.281 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:49.281 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:49.281 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:49.281 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:49.281 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:49.281 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:49.281 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:49.282 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:49.282 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:49.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f7023fff640 4 set_mon_vals no callback set 2024-03-28T16:15:49.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f703415c640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:49.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f703415c640 10 monclient: _renew_subs 2024-03-28T16:15:49.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f703415c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:49.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f703415c640 1 -- 172.21.15.7:0/2526148944 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f702c1404b0 con 0x7f702c0b32e0 2024-03-28T16:15:49.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f703415c640 1 librados: init done 2024-03-28T16:15:49.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.277+0000 7f703415c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:49.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.278+0000 7f703415c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:49.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.278+0000 7f703415c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:49.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.278+0000 7f703415c640 1 -- 172.21.15.7:0/2526148944 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f702c13e080 con 0x7f702c0b32e0 2024-03-28T16:15:49.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.295+0000 7f70227fc640 1 -- 172.21.15.7:0/2526148944 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f702403eae0 con 0x7f702c0b32e0 2024-03-28T16:15:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.295+0000 7f70227fc640 1 --2- 172.21.15.7:0/2526148944 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f701006c590 0x7f701006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.295+0000 7f7031ef8640 1 --2- 172.21.15.7:0/2526148944 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f701006c590 0x7f701006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.295+0000 7f7031ef8640 10 monclient: get_auth_request con 0x7f701006c590 auth_method 0 2024-03-28T16:15:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.296+0000 7f70227fc640 10 client.4294.objecter ms_handle_connect 0x7f701006c590 2024-03-28T16:15:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.296+0000 7f7031ef8640 1 --2- 172.21.15.7:0/2526148944 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f701006c590 0x7f701006ea50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f701c0017f0 tx=0x7f701c009040 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.296+0000 7f70227fc640 1 -- 172.21.15.7:0/2526148944 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f702407b840 con 0x7f702c0b32e0 2024-03-28T16:15:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.296+0000 7f70227fc640 10 client.4294.objecter ms_dispatch 0x7f702c000b90 osd_map(27..27 src has 1..27) v4 2024-03-28T16:15:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.296+0000 7f70227fc640 3 client.4294.objecter handle_osd_map got epochs [27,27] > 0 2024-03-28T16:15:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.296+0000 7f70227fc640 3 client.4294.objecter handle_osd_map decoding full epoch 27 2024-03-28T16:15:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.296+0000 7f70227fc640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-28T16:15:49.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.535+0000 7f70227fc640 1 -- 172.21.15.7:0/2526148944 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7024077b80 con 0x7f702c0b32e0 2024-03-28T16:15:49.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.535+0000 7f70227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:49.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.535+0000 7f70227fc640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:49.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.732+0000 7f703415c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:49.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.732+0000 7f703415c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:49.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.732+0000 7f703415c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:49.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.732+0000 7f703415c640 1 -- 172.21.15.7:0/2526148944 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f702c13e080 con 0x7f702c0b32e0 2024-03-28T16:15:49.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.809+0000 7f70227fc640 1 -- 172.21.15.7:0/2526148944 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+14098 (secure 0 0 0) 0x7f7024043090 con 0x7f702c0b32e0 2024-03-28T16:15:49.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.809+0000 7f70227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:49.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.809+0000 7f70227fc640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:49.810 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:15:49.811 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"fsid":"808ff1a1-2b17-4c35-a89a-340dbbf674ee","created":"2024-03-28T16:14:44.591599+0000","modified":"2024-03-28T16:15:49.250155+0000","last_up_change":"2024-03-28T16:15:40.088013+0000","last_in_change":"2024-03-28T16:14:49.757700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T16:15:31.373047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-28T16:15:45.986355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a19c6856-78b9-4919-97ce-6940a5af982c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":34918},{"type":"v1","addr":"172.21.15.7:6809","nonce":34918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":34918},{"type":"v1","addr":"172.21.15.7:6811","nonce":34918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":34918},{"type":"v1","addr":"172.21.15.7:6815","nonce":34918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":34918},{"type":"v1","addr":"172.21.15.7:6813","nonce":34918}]},"public_addr":"172.21.15.7:6809/34918","cluster_addr":"172.21.15.7:6811/34918","heartbeat_back_addr":"172.21.15.7:6815/34918","heartbeat_front_addr":"172.21.15.7:6813/34918","state":["exists","up"]},{"osd":1,"uuid":"2afee73d-3369-4fd2-bfc5-36dee610daf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":34919},{"type":"v1","addr":"172.21.15.7:6801","nonce":34919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":34919},{"type":"v1","addr":"172.21.15.7:6803","nonce":34919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":34919},{"type":"v1","addr":"172.21.15.7:6807","nonce":34919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":34919},{"type":"v1","addr":"172.21.15.7:6805","nonce":34919}]},"public_addr":"172.21.15.7:6801/34919","cluster_addr":"172.21.15.7:6803/34919","heartbeat_back_addr":"172.21.15.7:6807/34919","heartbeat_front_addr":"172.21.15.7:6805/34919","state":["exists","up"]},{"osd":2,"uuid":"0473d84c-e864-4373-b9d1-283130278363","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":34920},{"type":"v1","addr":"172.21.15.7:6825","nonce":34920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":34920},{"type":"v1","addr":"172.21.15.7:6827","nonce":34920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":34920},{"type":"v1","addr":"172.21.15.7:6831","nonce":34920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":34920},{"type":"v1","addr":"172.21.15.7:6829","nonce":34920}]},"public_addr":"172.21.15.7:6825/34920","cluster_addr":"172.21.15.7:6827/34920","heartbeat_back_addr":"172.21.15.7:6831/34920","heartbeat_front_addr":"172.21.15.7:6829/34920","state":["exists","up"]},{"osd":3,"uuid":"b64fd5e7-cdec-4bd0-b5e2-dfc8d4f6b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":34921},{"type":"v1","addr":"172.21.15.7:6817","nonce":34921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":34921},{"type":"v1","addr":"172.21.15.7:6819","nonce":34921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":34921},{"type":"v1","addr":"172.21.15.7:6823","nonce":34921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":34921},{"type":"v1","addr":"172.21.15.7:6821","nonce":34921}]},"public_addr":"172.21.15.7:6817/34921","cluster_addr":"172.21.15.7:6819/34921","heartbeat_back_addr":"172.21.15.7:6823/34921","heartbeat_front_addr":"172.21.15.7:6821/34921","state":["exists","up"]},{"osd":4,"uuid":"8f7552db-5651-430f-a826-3c6843bffb99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":35294},{"type":"v1","addr":"172.21.15.83:6819","nonce":35294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":35294},{"type":"v1","addr":"172.21.15.83:6821","nonce":35294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":35294},{"type":"v1","addr":"172.21.15.83:6825","nonce":35294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":35294},{"type":"v1","addr":"172.21.15.83:6823","nonce":35294}]},"public_addr":"172.21.15.83:6819/35294","cluster_addr":"172.21.15.83:6821/35294","heartbeat_back_addr":"172.21.15.83:6825/35294","heartbeat_front_addr":"172.21.15.83:6823/35294","state":["exists","up"]},{"osd":5,"uuid":"2aabdbd2-d870-4d6d-823e-89b008ea3108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":35296},{"type":"v1","addr":"172.21.15.83:6827","nonce":35296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":35296},{"type":"v1","addr":"172.21.15.83:6829","nonce":35296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":35296},{"type":"v1","addr":"172.21.15.83:6833","nonce":35296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":35296},{"type":"v1","addr":"172.21.15.83:6831","nonce":35296}]},"public_addr":"172.21.15.83:6827/35296","cluster_addr":"172.21.15.83:6829/35296","heartbeat_back_addr":"172.21.15.83:6833/35296","heartbeat_front_addr":"172.21.15.83:6831/35296","state":["exists","up"]},{"osd":6,"uuid":"fbbb7557-08d4-48d4-9b88-6bb1d680a87d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":35297},{"type":"v1","addr":"172.21.15.83:6811","nonce":35297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":35297},{"type":"v1","addr":"172.21.15.83:6813","nonce":35297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":35297},{"type":"v1","addr":"172.21.15.83:6817","nonce":35297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":35297},{"type":"v1","addr":"172.21.15.83:6815","nonce":35297}]},"public_addr":"172.21.15.83:6811/35297","cluster_addr":"172.21.15.83:6813/35297","heartbeat_back_addr":"172.21.15.83:6817/35297","heartbeat_front_addr":"172.21.15.83:6815/35297","state":["exists","up"]},{"osd":7,"uuid":"805f54d9-03d5-4b23-aef3-e1d55bb09f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":35295},{"type":"v1","addr":"172.21.15.83:6803","nonce":35295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":35295},{"type":"v1","addr":"172.21.15.83:6805","nonce":35295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":35295},{"type":"v1","addr":"172.21.15.83:6809","nonce":35295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":35295},{"type":"v1","addr":"172.21.15.83:6807","nonce":35295}]},"public_addr":"172.21.15.83:6803/35295","cluster_addr":"172.21.15.83:6805/35295","heartbeat_back_addr":"172.21.15.83:6809/35295","heartbeat_front_addr":"172.21.15.83:6807/35295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:27.198013+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:28.569762+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:28.085669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:28.165751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:26.831000+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:28.235232+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:27.056704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:26.424892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:15:49.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.811+0000 7f703415c640 10 librados: watch_flush enter 2024-03-28T16:15:49.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.811+0000 7f703415c640 10 librados: watch_flush exit 2024-03-28T16:15:49.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.811+0000 7f703415c640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-28T16:15:49.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.811+0000 7f703415c640 10 client.4294.objecter successfully canceled tick 2024-03-28T16:15:49.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.811+0000 7f703415c640 1 -- 172.21.15.7:0/2526148944 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f701006c590 msgr2=0x7f701006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:49.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.811+0000 7f703415c640 1 --2- 172.21.15.7:0/2526148944 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f701006c590 0x7f701006ea50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f701c0017f0 tx=0x7f701c009040 comp rx=0 tx=0).stop 2024-03-28T16:15:49.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.811+0000 7f703415c640 10 monclient: shutdown 2024-03-28T16:15:49.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.811+0000 7f703415c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:49.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.811+0000 7f703415c640 1 -- 172.21.15.7:0/2526148944 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f702c0b32e0 msgr2=0x7f702c1431c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:49.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.811+0000 7f703415c640 1 --2- 172.21.15.7:0/2526148944 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f702c0b32e0 0x7f702c1431c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7024032000 tx=0x7f7024004a40 comp rx=0 tx=0).stop 2024-03-28T16:15:49.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.811+0000 7f703415c640 1 -- 172.21.15.7:0/2526148944 shutdown_connections 2024-03-28T16:15:49.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.811+0000 7f703415c640 1 --2- 172.21.15.7:0/2526148944 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f701006c590 0x7f701006ea50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:49.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.811+0000 7f703415c640 1 --2- 172.21.15.7:0/2526148944 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f702c0b32e0 0x7f702c1431c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:49.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.811+0000 7f703415c640 1 --2- 172.21.15.7:0/2526148944 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f702c0a86e0 0x7f702c142c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:49.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.811+0000 7f703415c640 1 --2- 172.21.15.7:0/2526148944 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f702c071480 0x7f702c13d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:49.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.811+0000 7f703415c640 1 -- 172.21.15.7:0/2526148944 >> 172.21.15.7:0/2526148944 conn(0x7f702c0b70e0 msgr2=0x7f702c0b4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:49.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.811+0000 7f703415c640 1 -- 172.21.15.7:0/2526148944 shutdown_connections 2024-03-28T16:15:49.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.811+0000 7f703415c640 1 -- 172.21.15.7:0/2526148944 wait complete. 2024-03-28T16:15:49.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.811+0000 7f703415c640 1 librados: shutdown 2024-03-28T16:15:49.824 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-28T16:15:49.824 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:15:49.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.925+0000 7f90f7fd2640 10 monclient: get_monmap_and_config 2024-03-28T16:15:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.925+0000 7f90f7fd2640 10 monclient: build_initial_monmap 2024-03-28T16:15:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.925+0000 7f90f7fd2640 10 monclient: monmap: 2024-03-28T16:15:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:49.927421+0000 2024-03-28T16:15:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:49.927421+0000 2024-03-28T16:15:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:49.927 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:49.927 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:49.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.926+0000 7f90f7fd2640 1 Processor -- start 2024-03-28T16:15:49.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.926+0000 7f90f7fd2640 1 -- start start 2024-03-28T16:15:49.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.926+0000 7f90f7fd2640 10 monclient: init 2024-03-28T16:15:49.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.927+0000 7f90f7fd2640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:49.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.927+0000 7f90f7fd2640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-03-28T16:15:49.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.927+0000 7f90f7fd2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f90f006fad0 0x7f90f013f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:49.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.927+0000 7f90f7fd2640 10 monclient(hunting): picked mon.noname-c con 0x7f90f006fad0 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:49.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.927+0000 7f90f7fd2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90f006feb0 0x7f90f013fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:49.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.927+0000 7f90f7fd2640 10 monclient(hunting): picked mon.noname-b con 0x7f90f006feb0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:49.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.927+0000 7f90f7fd2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f90f013c4e0 addr v1:172.21.15.83:6789/0 2024-03-28T16:15:49.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.927+0000 7f90f7fd2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90f013f2e0 con 0x7f90f013c4e0 2024-03-28T16:15:49.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.927+0000 7f90f7fd2640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:49.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.927+0000 7f90f7fd2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f01401f0 con 0x7f90f006fad0 2024-03-28T16:15:49.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.927+0000 7f90f7fd2640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:49.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.927+0000 7f90f7fd2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f00afd50 con 0x7f90f006feb0 2024-03-28T16:15:49.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.927+0000 7f90f7fd2640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:49.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.927+0000 7f90f7fd2640 10 monclient(hunting): authenticate will time out at 1417.069082s 2024-03-28T16:15:49.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.927+0000 7f90f556d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90f006feb0 0x7f90f013fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.927+0000 7f90f5d6e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f90f006fad0 0x7f90f013f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.928+0000 7f90f556d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90f006feb0 0x7f90f013fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60072/0 (socket says 172.21.15.7:60072) 2024-03-28T16:15:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.928+0000 7f90f556d640 1 -- 172.21.15.7:0/194579150 learned_addr learned my addr 172.21.15.7:0/194579150 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.928+0000 7f90f556d640 10 monclient(hunting): get_auth_request con 0x7f90f006feb0 auth_method 0 2024-03-28T16:15:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.928+0000 7f90f556d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.928+0000 7f90f556d640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.928+0000 7f90f556d640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.928+0000 7f90f5d6e640 10 monclient(hunting): get_auth_request con 0x7f90f006fad0 auth_method 0 2024-03-28T16:15:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.928+0000 7f90f5d6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.928+0000 7f90f5d6e640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.928+0000 7f90f5d6e640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.928+0000 7f90f556d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.928+0000 7f90f556d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.928+0000 7f90f556d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.928+0000 7f90f5d6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:49.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.928+0000 7f90f5d6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:49.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.928+0000 7f90f5d6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:49.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.930+0000 7f90f5d6e640 10 monclient(hunting): handle_auth_done global_id 4297 payload 274 2024-03-28T16:15:49.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.930+0000 7f90f5d6e640 1 -- 172.21.15.7:0/194579150 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90f006feb0 msgr2=0x7f90f013fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:49.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.930+0000 7f90f5d6e640 1 --2- 172.21.15.7:0/194579150 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90f006feb0 0x7f90f013fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:49.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.930+0000 7f90f5d6e640 1 -- 172.21.15.7:0/194579150 >> v1:172.21.15.83:6789/0 conn(0x7f90f013c4e0 legacy=0x7f90f013c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:49.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.930+0000 7f90f5d6e640 10 monclient: _finish_hunting 0 2024-03-28T16:15:49.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.930+0000 7f90f5d6e640 1 monclient: found mon.noname-c 2024-03-28T16:15:49.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.930+0000 7f90f5d6e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:49.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.930+0000 7f90f5d6e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3300/0 2024-03-28T16:15:49.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.930+0000 7f90f5d6e640 1 -- 172.21.15.7:0/194579150 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f00717d0 con 0x7f90f006fad0 2024-03-28T16:15:49.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.930+0000 7f90f556d640 1 --2- 172.21.15.7:0/194579150 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90f006feb0 0x7f90f013fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:15:49.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.933+0000 7f90f5d6e640 1 --2- 172.21.15.7:0/194579150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90f006fad0 0x7f90f013f770 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f90f0140330 tx=0x7f90ec02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81039bd7a3cca2ac server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.936+0000 7f90f4d6c640 1 -- 172.21.15.7:0/194579150 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ec0316f0 con 0x7f90f006fad0 2024-03-28T16:15:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f4d6c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f4d6c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f4d6c640 10 monclient: dump: 2024-03-28T16:15:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:49.940 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:49.940 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:49.940 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:49.940 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:49.940 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:49.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f4d6c640 10 monclient: _finish_auth 0 2024-03-28T16:15:49.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f4d6c640 10 monclient: _check_auth_tickets 2024-03-28T16:15:49.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f4d6c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:49.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f4d6c640 1 -- 172.21.15.7:0/194579150 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ec031850 con 0x7f90f006fad0 2024-03-28T16:15:49.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f4d6c640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:49.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f4d6c640 1 -- 172.21.15.7:0/194579150 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ec031be0 con 0x7f90f006fad0 2024-03-28T16:15:49.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f7fd2640 5 monclient: authenticate success, global_id 4297 2024-03-28T16:15:49.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f7fd2640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:49.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f7fd2640 4 set_mon_vals no callback set 2024-03-28T16:15:49.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f4d6c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:49.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f4d6c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:49.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f4d6c640 10 monclient: dump: 2024-03-28T16:15:49.941 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:49.941 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:49.941 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:49.941 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:49.941 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:49.941 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:49.941 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:49.942 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:49.942 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:49.942 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:49.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f7fd2640 10 monclient: shutdown 2024-03-28T16:15:49.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f7fd2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:49.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f7fd2640 1 -- 172.21.15.7:0/194579150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90f006fad0 msgr2=0x7f90f013f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:49.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f7fd2640 1 --2- 172.21.15.7:0/194579150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90f006fad0 0x7f90f013f770 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f90f0140330 tx=0x7f90ec02b790 comp rx=0 tx=0).stop 2024-03-28T16:15:49.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f7fd2640 1 -- 172.21.15.7:0/194579150 shutdown_connections 2024-03-28T16:15:49.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f7fd2640 1 --2- 172.21.15.7:0/194579150 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90f006feb0 0x7f90f013fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:49.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f7fd2640 1 --2- 172.21.15.7:0/194579150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90f006fad0 0x7f90f013f770 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:49.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.937+0000 7f90f7fd2640 1 -- 172.21.15.7:0/194579150 >> 172.21.15.7:0/194579150 conn(0x7f90f00b5220 msgr2=0x7f90f00b5630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:49.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.938+0000 7f90f7fd2640 1 -- 172.21.15.7:0/194579150 shutdown_connections 2024-03-28T16:15:49.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.938+0000 7f90f7fd2640 1 -- 172.21.15.7:0/194579150 wait complete. 2024-03-28T16:15:49.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.938+0000 7f90f7fd2640 10 monclient: build_initial_monmap 2024-03-28T16:15:49.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.938+0000 7f90f7fd2640 10 monclient: monmap: 2024-03-28T16:15:49.945 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:49.946 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:49.946 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:49.946 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:49.946 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:49.946 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:49.946 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:49.946 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:49.946 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:49.946 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:49.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.938+0000 7f90f7fd2640 1 librados: starting msgr at 2024-03-28T16:15:49.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.938+0000 7f90f7fd2640 1 librados: starting objecter 2024-03-28T16:15:49.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.939+0000 7f90f7fd2640 1 Processor -- start 2024-03-28T16:15:49.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.939+0000 7f90f7fd2640 1 -- start start 2024-03-28T16:15:49.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.939+0000 7f90f7fd2640 1 librados: setting wanted keys 2024-03-28T16:15:49.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.939+0000 7f90f7fd2640 1 librados: calling monclient init 2024-03-28T16:15:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.939+0000 7f90f7fd2640 10 monclient: init 2024-03-28T16:15:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.939+0000 7f90f7fd2640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.939+0000 7f90f7fd2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-28T16:15:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.939+0000 7f90f7fd2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90f006fad0 0x7f90f00b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.939+0000 7f90f7fd2640 10 monclient(hunting): picked mon.noname-a con 0x7f90f006fad0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.939+0000 7f90f7fd2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90f006feb0 0x7f90f0069ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.939+0000 7f90f7fd2640 10 monclient(hunting): picked mon.noname-c con 0x7f90f006feb0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.939+0000 7f90f7fd2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90f013c4e0 0x7f90f006aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.939+0000 7f90f7fd2640 10 monclient(hunting): picked mon.noname-b con 0x7f90f013c4e0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.939+0000 7f90f7fd2640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.939+0000 7f90f7fd2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f012f630 con 0x7f90f006fad0 2024-03-28T16:15:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.939+0000 7f90f7fd2640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.939+0000 7f90f7fd2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f006a590 con 0x7f90f013c4e0 2024-03-28T16:15:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.939+0000 7f90f7fd2640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.939+0000 7f90f7fd2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f012f880 con 0x7f90f006feb0 2024-03-28T16:15:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.939+0000 7f90f7fd2640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.939+0000 7f90f7fd2640 10 monclient(hunting): authenticate will time out at 1417.081438s 2024-03-28T16:15:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f556d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90f006feb0 0x7f90f0069ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f656f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90f013c4e0 0x7f90f006aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f5d6e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90f006fad0 0x7f90f00b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f556d640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90f006feb0 0x7f90f0069ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60082/0 (socket says 172.21.15.7:60082) 2024-03-28T16:15:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f556d640 1 -- 172.21.15.7:0/14827662 learned_addr learned my addr 172.21.15.7:0/14827662 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f556d640 10 monclient(hunting): get_auth_request con 0x7f90f006feb0 auth_method 0 2024-03-28T16:15:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f556d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f556d640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f556d640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f656f640 10 monclient(hunting): get_auth_request con 0x7f90f013c4e0 auth_method 0 2024-03-28T16:15:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f656f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f656f640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f656f640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f5d6e640 10 monclient(hunting): get_auth_request con 0x7f90f006fad0 auth_method 0 2024-03-28T16:15:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f5d6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f5d6e640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f5d6e640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f556d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f556d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f556d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f656f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f656f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.940+0000 7f90f656f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.941+0000 7f90f5d6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.941+0000 7f90f5d6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.941+0000 7f90f5d6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.943+0000 7f90f656f640 10 monclient(hunting): handle_auth_done global_id 4300 payload 824 2024-03-28T16:15:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.943+0000 7f90f656f640 1 -- 172.21.15.7:0/14827662 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90f006feb0 msgr2=0x7f90f0069ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.943+0000 7f90f656f640 1 --2- 172.21.15.7:0/14827662 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90f006feb0 0x7f90f0069ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:49.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.943+0000 7f90f656f640 1 -- 172.21.15.7:0/14827662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90f006fad0 msgr2=0x7f90f00b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:49.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.943+0000 7f90f656f640 1 --2- 172.21.15.7:0/14827662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90f006fad0 0x7f90f00b45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:49.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.943+0000 7f90f656f640 10 monclient: _finish_hunting 0 2024-03-28T16:15:49.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.943+0000 7f90f656f640 1 monclient: found mon.noname-b 2024-03-28T16:15:49.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.943+0000 7f90f656f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:49.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.943+0000 7f90f656f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:15:49.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.943+0000 7f90f656f640 1 -- 172.21.15.7:0/14827662 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e00094d0 con 0x7f90f013c4e0 2024-03-28T16:15:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.945+0000 7f90f656f640 1 --2- 172.21.15.7:0/14827662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90f013c4e0 0x7f90f006aab0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f90e800c780 tx=0x7f90e800cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.945+0000 7f90e67fc640 10 client.?.objecter ms_handle_connect 0x7f90f013c4e0 2024-03-28T16:15:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.945+0000 7f90e67fc640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.948+0000 7f90e67fc640 1 -- 172.21.15.7:0/14827662 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e8003b70 con 0x7f90f013c4e0 2024-03-28T16:15:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.948+0000 7f90e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.948+0000 7f90e67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.948+0000 7f90e67fc640 10 monclient: dump: 2024-03-28T16:15:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:49.952 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.948+0000 7f90e67fc640 10 monclient: _finish_auth 0 2024-03-28T16:15:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.948+0000 7f90e67fc640 10 monclient: _check_auth_tickets 2024-03-28T16:15:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.948+0000 7f90e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.948+0000 7f90e67fc640 1 -- 172.21.15.7:0/14827662 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e8003cf0 con 0x7f90f013c4e0 2024-03-28T16:15:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.948+0000 7f90e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.948+0000 7f90e67fc640 1 -- 172.21.15.7:0/14827662 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e8003ea0 con 0x7f90f013c4e0 2024-03-28T16:15:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.948+0000 7f90e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.948+0000 7f90e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.948+0000 7f90e67fc640 10 monclient: dump: 2024-03-28T16:15:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:49.954 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.948+0000 7f90f7fd2640 5 monclient: authenticate success, global_id 4300 2024-03-28T16:15:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.948+0000 7f90e7fff640 4 set_mon_vals no callback set 2024-03-28T16:15:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.948+0000 7f90f7fd2640 10 monclient: _renew_subs 2024-03-28T16:15:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.948+0000 7f90f7fd2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.948+0000 7f90f7fd2640 1 -- 172.21.15.7:0/14827662 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90ec009400 con 0x7f90f013c4e0 2024-03-28T16:15:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.949+0000 7f90f7fd2640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.949+0000 7f90f7fd2640 10 monclient: _renew_subs 2024-03-28T16:15:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.949+0000 7f90f7fd2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.949+0000 7f90f7fd2640 1 -- 172.21.15.7:0/14827662 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f01d9fa0 con 0x7f90f013c4e0 2024-03-28T16:15:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.949+0000 7f90f7fd2640 1 librados: init done 2024-03-28T16:15:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.949+0000 7f90f7fd2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.949+0000 7f90f7fd2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.949+0000 7f90f7fd2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.949+0000 7f90f7fd2640 1 -- 172.21.15.7:0/14827662 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f006a400 con 0x7f90f013c4e0 2024-03-28T16:15:49.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.967+0000 7f90e67fc640 1 -- 172.21.15.7:0/14827662 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f90e80052b0 con 0x7f90f013c4e0 2024-03-28T16:15:49.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.967+0000 7f90e67fc640 1 --2- 172.21.15.7:0/14827662 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f90d406c5e0 0x7f90d406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.968+0000 7f90f5d6e640 1 --2- 172.21.15.7:0/14827662 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f90d406c5e0 0x7f90d406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.968+0000 7f90f5d6e640 10 monclient: get_auth_request con 0x7f90d406c5e0 auth_method 0 2024-03-28T16:15:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.968+0000 7f90f5d6e640 1 --2- 172.21.15.7:0/14827662 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f90d406c5e0 0x7f90d406eaa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f90ec036000 tx=0x7f90ec008030 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.968+0000 7f90e67fc640 10 client.4300.objecter ms_handle_connect 0x7f90d406c5e0 2024-03-28T16:15:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.969+0000 7f90e67fc640 1 -- 172.21.15.7:0/14827662 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f90e8057ef0 con 0x7f90f013c4e0 2024-03-28T16:15:49.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.969+0000 7f90e67fc640 10 client.4300.objecter ms_dispatch 0x7f90f0000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:49.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.969+0000 7f90e67fc640 3 client.4300.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:49.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.969+0000 7f90e67fc640 3 client.4300.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:49.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:49.969+0000 7f90e67fc640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-28T16:15:50.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.213+0000 7f90e67fc640 1 -- 172.21.15.7:0/14827662 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f90e8054230 con 0x7f90f013c4e0 2024-03-28T16:15:50.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.213+0000 7f90e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:50.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.213+0000 7f90e67fc640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:50.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.399+0000 7f90f7fd2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:50.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.399+0000 7f90f7fd2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:50.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.399+0000 7f90f7fd2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:50.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.399+0000 7f90f7fd2640 1 -- 172.21.15.7:0/14827662 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90f006a7d0 con 0x7f90f013c4e0 2024-03-28T16:15:50.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.425+0000 7f90e67fc640 1 -- 172.21.15.7:0/14827662 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+14098 (secure 0 0 0) 0x7f90e8057850 con 0x7f90f013c4e0 2024-03-28T16:15:50.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.425+0000 7f90e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-28T16:15:50.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.425+0000 7f90e67fc640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:50.426 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:15:50.426 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"fsid":"808ff1a1-2b17-4c35-a89a-340dbbf674ee","created":"2024-03-28T16:14:44.591599+0000","modified":"2024-03-28T16:15:49.250155+0000","last_up_change":"2024-03-28T16:15:40.088013+0000","last_in_change":"2024-03-28T16:14:49.757700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T16:15:31.373047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-28T16:15:45.986355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a19c6856-78b9-4919-97ce-6940a5af982c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":34918},{"type":"v1","addr":"172.21.15.7:6809","nonce":34918}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":34918},{"type":"v1","addr":"172.21.15.7:6811","nonce":34918}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":34918},{"type":"v1","addr":"172.21.15.7:6815","nonce":34918}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":34918},{"type":"v1","addr":"172.21.15.7:6813","nonce":34918}]},"public_addr":"172.21.15.7:6809/34918","cluster_addr":"172.21.15.7:6811/34918","heartbeat_back_addr":"172.21.15.7:6815/34918","heartbeat_front_addr":"172.21.15.7:6813/34918","state":["exists","up"]},{"osd":1,"uuid":"2afee73d-3369-4fd2-bfc5-36dee610daf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":34919},{"type":"v1","addr":"172.21.15.7:6801","nonce":34919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":34919},{"type":"v1","addr":"172.21.15.7:6803","nonce":34919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":34919},{"type":"v1","addr":"172.21.15.7:6807","nonce":34919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":34919},{"type":"v1","addr":"172.21.15.7:6805","nonce":34919}]},"public_addr":"172.21.15.7:6801/34919","cluster_addr":"172.21.15.7:6803/34919","heartbeat_back_addr":"172.21.15.7:6807/34919","heartbeat_front_addr":"172.21.15.7:6805/34919","state":["exists","up"]},{"osd":2,"uuid":"0473d84c-e864-4373-b9d1-283130278363","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":34920},{"type":"v1","addr":"172.21.15.7:6825","nonce":34920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":34920},{"type":"v1","addr":"172.21.15.7:6827","nonce":34920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":34920},{"type":"v1","addr":"172.21.15.7:6831","nonce":34920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":34920},{"type":"v1","addr":"172.21.15.7:6829","nonce":34920}]},"public_addr":"172.21.15.7:6825/34920","cluster_addr":"172.21.15.7:6827/34920","heartbeat_back_addr":"172.21.15.7:6831/34920","heartbeat_front_addr":"172.21.15.7:6829/34920","state":["exists","up"]},{"osd":3,"uuid":"b64fd5e7-cdec-4bd0-b5e2-dfc8d4f6b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":34921},{"type":"v1","addr":"172.21.15.7:6817","nonce":34921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":34921},{"type":"v1","addr":"172.21.15.7:6819","nonce":34921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":34921},{"type":"v1","addr":"172.21.15.7:6823","nonce":34921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":34921},{"type":"v1","addr":"172.21.15.7:6821","nonce":34921}]},"public_addr":"172.21.15.7:6817/34921","cluster_addr":"172.21.15.7:6819/34921","heartbeat_back_addr":"172.21.15.7:6823/34921","heartbeat_front_addr":"172.21.15.7:6821/34921","state":["exists","up"]},{"osd":4,"uuid":"8f7552db-5651-430f-a826-3c6843bffb99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":35294},{"type":"v1","addr":"172.21.15.83:6819","nonce":35294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":35294},{"type":"v1","addr":"172.21.15.83:6821","nonce":35294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":35294},{"type":"v1","addr":"172.21.15.83:6825","nonce":35294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":35294},{"type":"v1","addr":"172.21.15.83:6823","nonce":35294}]},"public_addr":"172.21.15.83:6819/35294","cluster_addr":"172.21.15.83:6821/35294","heartbeat_back_addr":"172.21.15.83:6825/35294","heartbeat_front_addr":"172.21.15.83:6823/35294","state":["exists","up"]},{"osd":5,"uuid":"2aabdbd2-d870-4d6d-823e-89b008ea3108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":35296},{"type":"v1","addr":"172.21.15.83:6827","nonce":35296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":35296},{"type":"v1","addr":"172.21.15.83:6829","nonce":35296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":35296},{"type":"v1","addr":"172.21.15.83:6833","nonce":35296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":35296},{"type":"v1","addr":"172.21.15.83:6831","nonce":35296}]},"public_addr":"172.21.15.83:6827/35296","cluster_addr":"172.21.15.83:6829/35296","heartbeat_back_addr":"172.21.15.83:6833/35296","heartbeat_front_addr":"172.21.15.83:6831/35296","state":["exists","up"]},{"osd":6,"uuid":"fbbb7557-08d4-48d4-9b88-6bb1d680a87d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":35297},{"type":"v1","addr":"172.21.15.83:6811","nonce":35297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":35297},{"type":"v1","addr":"172.21.15.83:6813","nonce":35297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":35297},{"type":"v1","addr":"172.21.15.83:6817","nonce":35297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":35297},{"type":"v1","addr":"172.21.15.83:6815","nonce":35297}]},"public_addr":"172.21.15.83:6811/35297","cluster_addr":"172.21.15.83:6813/35297","heartbeat_back_addr":"172.21.15.83:6817/35297","heartbeat_front_addr":"172.21.15.83:6815/35297","state":["exists","up"]},{"osd":7,"uuid":"805f54d9-03d5-4b23-aef3-e1d55bb09f87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":35295},{"type":"v1","addr":"172.21.15.83:6803","nonce":35295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":35295},{"type":"v1","addr":"172.21.15.83:6805","nonce":35295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":35295},{"type":"v1","addr":"172.21.15.83:6809","nonce":35295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":35295},{"type":"v1","addr":"172.21.15.83:6807","nonce":35295}]},"public_addr":"172.21.15.83:6803/35295","cluster_addr":"172.21.15.83:6805/35295","heartbeat_back_addr":"172.21.15.83:6809/35295","heartbeat_front_addr":"172.21.15.83:6807/35295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:27.198013+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:28.569762+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:28.085669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:28.165751+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:26.831000+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:28.235232+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:27.056704+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T16:15:26.424892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:15:50.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.426+0000 7f90f7fd2640 10 librados: watch_flush enter 2024-03-28T16:15:50.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.426+0000 7f90f7fd2640 10 librados: watch_flush exit 2024-03-28T16:15:50.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.426+0000 7f90f7fd2640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-28T16:15:50.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.426+0000 7f90f7fd2640 10 client.4300.objecter successfully canceled tick 2024-03-28T16:15:50.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.426+0000 7f90f7fd2640 1 -- 172.21.15.7:0/14827662 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f90d406c5e0 msgr2=0x7f90d406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.426+0000 7f90f7fd2640 1 --2- 172.21.15.7:0/14827662 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f90d406c5e0 0x7f90d406eaa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f90ec036000 tx=0x7f90ec008030 comp rx=0 tx=0).stop 2024-03-28T16:15:50.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.426+0000 7f90f7fd2640 10 monclient: shutdown 2024-03-28T16:15:50.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.426+0000 7f90f7fd2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:50.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.426+0000 7f90f7fd2640 1 -- 172.21.15.7:0/14827662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90f013c4e0 msgr2=0x7f90f006aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.426+0000 7f90f7fd2640 1 --2- 172.21.15.7:0/14827662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90f013c4e0 0x7f90f006aab0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f90e800c780 tx=0x7f90e800cc40 comp rx=0 tx=0).stop 2024-03-28T16:15:50.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.427+0000 7f90f7fd2640 1 -- 172.21.15.7:0/14827662 shutdown_connections 2024-03-28T16:15:50.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.427+0000 7f90f7fd2640 1 --2- 172.21.15.7:0/14827662 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f90d406c5e0 0x7f90d406eaa0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.427+0000 7f90f7fd2640 1 --2- 172.21.15.7:0/14827662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90f013c4e0 0x7f90f006aab0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.427+0000 7f90f7fd2640 1 --2- 172.21.15.7:0/14827662 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f90f006feb0 0x7f90f0069ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.427+0000 7f90f7fd2640 1 --2- 172.21.15.7:0/14827662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f90f006fad0 0x7f90f00b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.427+0000 7f90f7fd2640 1 -- 172.21.15.7:0/14827662 >> 172.21.15.7:0/14827662 conn(0x7f90f00b5220 msgr2=0x7f90f0139820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:50.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.427+0000 7f90f7fd2640 1 -- 172.21.15.7:0/14827662 shutdown_connections 2024-03-28T16:15:50.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.427+0000 7f90f7fd2640 1 -- 172.21.15.7:0/14827662 wait complete. 2024-03-28T16:15:50.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.427+0000 7f90f7fd2640 1 librados: shutdown 2024-03-28T16:15:50.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-28T16:15:50.440 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-28T16:15:50.440 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-28T16:15:50.441 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-28T16:15:50.441 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-28T16:15:50.441 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-28T16:15:50.442 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-28T16:15:50.442 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-28T16:15:50.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.613+0000 7f6c0ed37640 10 monclient: get_monmap_and_config 2024-03-28T16:15:50.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.614+0000 7f6c0ed37640 10 monclient: build_initial_monmap 2024-03-28T16:15:50.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.614+0000 7f6c0ed37640 10 monclient: monmap: 2024-03-28T16:15:50.616 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:50.616 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.616 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:50.615629+0000 2024-03-28T16:15:50.616 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:50.615629+0000 2024-03-28T16:15:50.616 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:50.616 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.616 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:50.617 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:50.617 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:50.617 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:50.617 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:50.617 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.616+0000 7fd314514640 10 monclient: get_monmap_and_config 2024-03-28T16:15:50.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.616+0000 7fd314514640 10 monclient: build_initial_monmap 2024-03-28T16:15:50.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.616+0000 7fd314514640 10 monclient: monmap: 2024-03-28T16:15:50.618 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:50.618 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.618 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:50.617790+0000 2024-03-28T16:15:50.618 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:50.617790+0000 2024-03-28T16:15:50.618 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:50.618 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.618 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:50.618 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:50.618 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:50.618 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:50.619 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.615+0000 7f075b037640 10 monclient: get_monmap_and_config 2024-03-28T16:15:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.615+0000 7f075b037640 10 monclient: build_initial_monmap 2024-03-28T16:15:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.615+0000 7f075b037640 10 monclient: monmap: 2024-03-28T16:15:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:50.617086+0000 2024-03-28T16:15:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:50.617086+0000 2024-03-28T16:15:50.620 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:50.620 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.620 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:50.620 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:50.620 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:50.620 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:50.620 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:50.620 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.615+0000 7f6c0ed37640 1 Processor -- start 2024-03-28T16:15:50.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.615+0000 7f6c0ed37640 1 -- start start 2024-03-28T16:15:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.615+0000 7f6c0ed37640 10 monclient: init 2024-03-28T16:15:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.616+0000 7f6c0ed37640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.616+0000 7f6c0ed37640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-03-28T16:15:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.616+0000 7f6c0ed37640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6c0002e1c0 0x7f6c0002c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.616+0000 7f6c0ed37640 10 monclient(hunting): picked mon.noname-b con 0x7f6c0002e1c0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.616+0000 7f6c0ed37640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6c0002ec00 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.616+0000 7f6c0ed37640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6c00032eb0 0x7f6c000cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.616+0000 7f6c0ed37640 10 monclient(hunting): picked mon.noname-c con 0x7f6c00032eb0 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.616+0000 7f6c0ed37640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c0002fba0 con 0x7f6c0002ec00 2024-03-28T16:15:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.616+0000 7f6c0ed37640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.616+0000 7f6c0ed37640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c0002d6e0 con 0x7f6c00032eb0 2024-03-28T16:15:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.616+0000 7f6c0ed37640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.616+0000 7f6c0ed37640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0002d820 con 0x7f6c0002e1c0 2024-03-28T16:15:50.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.616+0000 7f6c0ed37640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:50.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.616+0000 7f6c0ed37640 10 monclient(hunting): authenticate will time out at 1417.758701s 2024-03-28T16:15:50.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.616+0000 7f3473173640 10 monclient: get_monmap_and_config 2024-03-28T16:15:50.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.616+0000 7f3473173640 10 monclient: build_initial_monmap 2024-03-28T16:15:50.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.616+0000 7f3473173640 10 monclient: monmap: 2024-03-28T16:15:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:50.618547+0000 2024-03-28T16:15:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:50.618547+0000 2024-03-28T16:15:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:50.623 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.617+0000 7f075b037640 1 Processor -- start 2024-03-28T16:15:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.617+0000 7f075b037640 1 -- start start 2024-03-28T16:15:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.617+0000 7f075b037640 10 monclient: init 2024-03-28T16:15:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.617+0000 7febca622640 10 monclient: get_monmap_and_config 2024-03-28T16:15:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.617+0000 7febca622640 10 monclient: build_initial_monmap 2024-03-28T16:15:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.617+0000 7febca622640 10 monclient: monmap: 2024-03-28T16:15:50.624 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:50.625 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.625 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:50.619103+0000 2024-03-28T16:15:50.625 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:50.619103+0000 2024-03-28T16:15:50.625 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:50.625 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.625 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:50.625 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:50.625 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:50.625 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:50.625 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:50.625 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.618+0000 7fd314514640 1 Processor -- start 2024-03-28T16:15:50.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.618+0000 7f075b037640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:50.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.618+0000 7f075b037640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-03-28T16:15:50.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.618+0000 7f075b037640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f075413d3e0 addr v1:172.21.15.83:6789/0 2024-03-28T16:15:50.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.618+0000 7f075b037640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f075413e000 0x7f075413d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.618+0000 7f075b037640 10 monclient(hunting): picked mon.noname-a con 0x7f075413e000 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:50.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.618+0000 7f075b037640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07540ada00 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:50.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7f075b037640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07540af0f0 con 0x7f07540ada00 2024-03-28T16:15:50.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7f075b037640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f075413afc0 con 0x7f075413d3e0 2024-03-28T16:15:50.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.618+0000 7f6c0d2d4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6c00032eb0 0x7f6c000cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.618+0000 7f6c0d2d4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6c00032eb0 0x7f6c000cc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:57954/0 (socket says 172.21.15.7:57954) 2024-03-28T16:15:50.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.618+0000 7f6c0d2d4640 1 -- 172.21.15.7:0/3234525566 learned_addr learned my addr 172.21.15.7:0/3234525566 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:50.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.618+0000 7f6c0d2d4640 10 monclient(hunting): get_auth_request con 0x7f6c00032eb0 auth_method 0 2024-03-28T16:15:50.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.618+0000 7f6c0d2d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.618+0000 7f6c0d2d4640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.618+0000 7f6c0d2d4640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7f075b037640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.618+0000 7fd314514640 1 -- start start 2024-03-28T16:15:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.618+0000 7fd314514640 10 monclient: init 2024-03-28T16:15:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7fd314514640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7fd314514640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-03-28T16:15:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7fd314514640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd30c0a8b60 0x7fd30c0a8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7fd314514640 10 monclient(hunting): picked mon.noname-c con 0x7fd30c0a8b60 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7fd314514640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd30c0a9dc0 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7fd314514640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd30c0ab4b0 0x7fd30c149440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7fd314514640 10 monclient(hunting): picked mon.noname-a con 0x7fd30c0ab4b0 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:50.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7fd314514640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd30c148cd0 con 0x7fd30c0a9dc0 2024-03-28T16:15:50.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7fd314514640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7fd314514640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd30c0ab960 con 0x7fd30c0ab4b0 2024-03-28T16:15:50.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7fd314514640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7fd314514640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd30c0ab090 con 0x7fd30c0a8b60 2024-03-28T16:15:50.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7fd314514640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:50.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7fd314514640 10 monclient(hunting): authenticate will time out at 1417.761342s 2024-03-28T16:15:50.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7f075b037640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07540ab1e0 con 0x7f075413e000 2024-03-28T16:15:50.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7f075b037640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:50.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7f075b037640 10 monclient(hunting): authenticate will time out at 1417.761653s 2024-03-28T16:15:50.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7f3473173640 1 Processor -- start 2024-03-28T16:15:50.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7f0758dd3640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f075413d3e0 0x7f075413a8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.7:42348/0 (socket says 172.21.15.7:42348) 2024-03-28T16:15:50.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7f0758dd3640 1 -- 172.21.15.7:0/2172790457 learned_addr learned my addr 172.21.15.7:0/2172790457 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-28T16:15:50.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.618+0000 7febca622640 1 Processor -- start 2024-03-28T16:15:50.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.618+0000 7febca622640 1 -- start start 2024-03-28T16:15:50.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.618+0000 7febca622640 10 monclient: init 2024-03-28T16:15:50.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.619+0000 7febca622640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:50.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7febca622640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-03-28T16:15:50.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7febca622640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7febc41408d0 addr v1:172.21.15.83:6789/0 2024-03-28T16:15:50.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7febca622640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7febc4140cb0 0x7febc4141380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7febca622640 10 monclient(hunting): picked mon.noname-a con 0x7febc4140cb0 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:50.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7f5fa6ec7640 10 monclient: get_monmap_and_config 2024-03-28T16:15:50.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7f5fa6ec7640 10 monclient: build_initial_monmap 2024-03-28T16:15:50.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7f5fa6ec7640 10 monclient: monmap: 2024-03-28T16:15:50.634 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:50.634 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.635 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:50.621751+0000 2024-03-28T16:15:50.635 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:50.621751+0000 2024-03-28T16:15:50.635 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:50.635 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.635 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:50.635 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:50.635 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:50.635 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:50.635 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:50.635 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7febca622640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7febc40ab950 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:50.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7febca622640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc413afc0 con 0x7febc40ab950 2024-03-28T16:15:50.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7febca622640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc40aa420 con 0x7febc41408d0 2024-03-28T16:15:50.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7f6c0d2d4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7febca622640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7f6c0d2d4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7febca622640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febc40ab250 con 0x7febc4140cb0 2024-03-28T16:15:50.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7febca622640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:50.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7febca622640 10 monclient(hunting): authenticate will time out at 1417.761951s 2024-03-28T16:15:50.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7f6c0d2d4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:50.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7f6c0cad3640 1 --2- 172.21.15.7:0/3234525566 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6c0002e1c0 0x7f6c0002c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7f6c0cad3640 10 monclient(hunting): get_auth_request con 0x7f6c0002e1c0 auth_method 0 2024-03-28T16:15:50.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7f6c0cad3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7f6c0cad3640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7f6c0cad3640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7f3473173640 1 -- start start 2024-03-28T16:15:50.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.620+0000 7f3473173640 10 monclient: init 2024-03-28T16:15:50.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.621+0000 7f5fa6ec7640 1 Processor -- start 2024-03-28T16:15:50.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.621+0000 7f5fa6ec7640 1 -- start start 2024-03-28T16:15:50.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.621+0000 7f5fa6ec7640 10 monclient: init 2024-03-28T16:15:50.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.621+0000 7f3473173640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:50.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.621+0000 7f3473173640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-03-28T16:15:50.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.621+0000 7f3473173640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f346c0ad840 0x7f346c13b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.621+0000 7f3473173640 10 monclient(hunting): picked mon.noname-c con 0x7f346c0ad840 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:50.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.621+0000 7f3473173640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f346c06f780 0x7f346c070e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.621+0000 7f3473173640 10 monclient(hunting): picked mon.noname-b con 0x7f346c06f780 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:50.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f3473173640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f346c13bac0 addr v1:172.21.15.83:6789/0 2024-03-28T16:15:50.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f3473173640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346c06e8d0 con 0x7f346c13bac0 2024-03-28T16:15:50.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f3473173640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f3473173640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f346c0adc20 con 0x7f346c0ad840 2024-03-28T16:15:50.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f3473173640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f3473173640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346c06f330 con 0x7f346c06f780 2024-03-28T16:15:50.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f3473173640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:50.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f3473173640 10 monclient(hunting): authenticate will time out at 1417.763870s 2024-03-28T16:15:50.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f51cdc08640 10 monclient: get_monmap_and_config 2024-03-28T16:15:50.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f51cdc08640 10 monclient: build_initial_monmap 2024-03-28T16:15:50.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f51cdc08640 10 monclient: monmap: 2024-03-28T16:15:50.641 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:50.641 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.641 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:50.623777+0000 2024-03-28T16:15:50.642 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:50.623777+0000 2024-03-28T16:15:50.642 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:50.642 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.642 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:50.642 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:50.642 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:50.642 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:50.642 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:50.642 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f5fa6ec7640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:50.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f6c0cad3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f6c0cad3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f5fa6ec7640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-03-28T16:15:50.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f6c0cad3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:50.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f5fa6ec7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f5fa006a490 0x7f5fa00712d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f5fa6ec7640 10 monclient(hunting): picked mon.noname-c con 0x7f5fa006a490 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:50.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f5fa6ec7640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5fa0071810 0x7f5fa0149000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f5fa6ec7640 10 monclient(hunting): picked mon.noname-b con 0x7f5fa0071810 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:50.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f5fa6ec7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5fa0149540 addr v1:172.21.15.83:6789/0 2024-03-28T16:15:50.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f5fa6ec7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa006ac40 con 0x7f5fa0149540 2024-03-28T16:15:50.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f5fa6ec7640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f5fa6ec7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa006fa30 con 0x7f5fa006a490 2024-03-28T16:15:50.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f5fa6ec7640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f5fa6ec7640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa0071c70 con 0x7f5fa0071810 2024-03-28T16:15:50.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f5fa6ec7640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:50.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.622+0000 7f5fa6ec7640 10 monclient(hunting): authenticate will time out at 1417.764382s 2024-03-28T16:15:50.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.623+0000 7f51cdc08640 1 Processor -- start 2024-03-28T16:15:50.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.623+0000 7f346bfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f346c06f780 0x7f346c070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.623+0000 7f51cdc08640 1 -- start start 2024-03-28T16:15:50.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.623+0000 7f51cdc08640 10 monclient: init 2024-03-28T16:15:50.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.623+0000 7f346bfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f346c06f780 0x7f346c070e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60102/0 (socket says 172.21.15.7:60102) 2024-03-28T16:15:50.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.623+0000 7f346bfff640 1 -- 172.21.15.7:0/3622774235 learned_addr learned my addr 172.21.15.7:0/3622774235 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:50.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.623+0000 7f346bfff640 10 monclient(hunting): get_auth_request con 0x7f346c06f780 auth_method 0 2024-03-28T16:15:50.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.623+0000 7f346bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.623+0000 7f346bfff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.623+0000 7f346bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f5fa56c4640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5fa0071810 0x7f5fa0149000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f07537fe640 1 -- 172.21.15.7:0/2172790457 <== mon.1 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288266759 0 0) 0x7f075413afc0 con 0x7f075413d3e0 2024-03-28T16:15:50.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f07537fe640 10 monclient(hunting): _negotiate 2024-03-28T16:15:50.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f07537fe640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f07537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f07537fe640 10 monclient(hunting): my global_id is 4306 2024-03-28T16:15:50.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f07537fe640 1 -- 172.21.15.7:0/2172790457 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f073c003520 con 0x7f075413d3e0 2024-03-28T16:15:50.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f51cdc08640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:50.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f51cdc08640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-03-28T16:15:50.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f51cdc08640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f51c80b3b90 0x7f51c80a86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f51cdc08640 10 monclient(hunting): picked mon.noname-c con 0x7f51c80b3b90 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:50.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f51cdc08640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f51c80b32e0 0x7f51c813e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f51cdc08640 10 monclient(hunting): picked mon.noname-a con 0x7f51c80b32e0 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:50.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f51cdc08640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f51c813eab0 0x7f51c81412d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f51cdc08640 10 monclient(hunting): picked mon.noname-b con 0x7f51c813eab0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:50.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f51cdc08640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f51cdc08640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51c80b3f70 con 0x7f51c80b32e0 2024-03-28T16:15:50.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f51cdc08640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f51cdc08640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51c80b1560 con 0x7f51c80b3b90 2024-03-28T16:15:50.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f51cdc08640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f51cdc08640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c80a8db0 con 0x7f51c813eab0 2024-03-28T16:15:50.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f51cdc08640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:50.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f51cdc08640 10 monclient(hunting): authenticate will time out at 1417.766300s 2024-03-28T16:15:50.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f346bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f346bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.624+0000 7f346bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:50.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.626+0000 7f6c0d2d4640 10 monclient(hunting): handle_auth_done global_id 4303 payload 274 2024-03-28T16:15:50.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.626+0000 7f6c0d2d4640 1 -- 172.21.15.7:0/3234525566 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6c0002e1c0 msgr2=0x7f6c0002c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:50.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.626+0000 7f6c0d2d4640 1 --2- 172.21.15.7:0/3234525566 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6c0002e1c0 0x7f6c0002c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.626+0000 7f6c0d2d4640 1 -- 172.21.15.7:0/3234525566 >> v1:172.21.15.7:6789/0 conn(0x7f6c0002ec00 legacy=0x7f6c000327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:50.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.626+0000 7f6c0d2d4640 10 monclient: _finish_hunting 0 2024-03-28T16:15:50.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.626+0000 7f6c0d2d4640 1 monclient: found mon.noname-c 2024-03-28T16:15:50.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.626+0000 7f6c0d2d4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:50.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.626+0000 7f6c0d2d4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3300/0 2024-03-28T16:15:50.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.626+0000 7f6c0d2d4640 1 -- 172.21.15.7:0/3234525566 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0002e5a0 con 0x7f6c00032eb0 2024-03-28T16:15:50.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.626+0000 7fd3122b0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd30c0a8b60 0x7fd30c0a8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.627+0000 7febc9620640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7febc41408d0 0x7febc413a8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.7:42358/0 (socket says 172.21.15.7:42358) 2024-03-28T16:15:50.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.627+0000 7febc9620640 1 -- 172.21.15.7:0/2217813657 learned_addr learned my addr 172.21.15.7:0/2217813657 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-28T16:15:50.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.627+0000 7fd3122b0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd30c0a8b60 0x7fd30c0a8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:57964/0 (socket says 172.21.15.7:57964) 2024-03-28T16:15:50.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.627+0000 7fd3122b0640 1 -- 172.21.15.7:0/3390582447 learned_addr learned my addr 172.21.15.7:0/3390582447 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:50.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.627+0000 7fd3122b0640 10 monclient(hunting): get_auth_request con 0x7fd30c0a8b60 auth_method 0 2024-03-28T16:15:50.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.627+0000 7fd3122b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.627+0000 7fd3122b0640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.627+0000 7fd3122b0640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.628+0000 7f5fa56c4640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5fa0071810 0x7f5fa0149000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60118/0 (socket says 172.21.15.7:60118) 2024-03-28T16:15:50.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.628+0000 7f5fa56c4640 1 -- 172.21.15.7:0/3513372345 learned_addr learned my addr 172.21.15.7:0/3513372345 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:50.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.628+0000 7f5fa56c4640 10 monclient(hunting): get_auth_request con 0x7f5fa0071810 auth_method 0 2024-03-28T16:15:50.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.628+0000 7f5fa56c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.628+0000 7f5fa56c4640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.628+0000 7f5fa56c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.629+0000 7f5fa5ec5640 1 --2- 172.21.15.7:0/3513372345 >> v2:172.21.15.83:3300/0 conn(0x7f5fa006a490 0x7f5fa00712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.629+0000 7f5fa5ec5640 10 monclient(hunting): get_auth_request con 0x7f5fa006a490 auth_method 0 2024-03-28T16:15:50.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.629+0000 7f5fa5ec5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.629+0000 7f5fa5ec5640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.629+0000 7f5fa5ec5640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.630+0000 7fd3122b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.630+0000 7fd3122b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.630+0000 7fd3122b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:50.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.630+0000 7f346bfff640 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2024-03-28T16:15:50.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.630+0000 7f346bfff640 1 -- 172.21.15.7:0/3622774235 >> v2:172.21.15.83:3300/0 conn(0x7f346c0ad840 msgr2=0x7f346c13b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:50.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.630+0000 7f346bfff640 1 --2- 172.21.15.7:0/3622774235 >> v2:172.21.15.83:3300/0 conn(0x7f346c0ad840 0x7f346c13b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.630+0000 7f346bfff640 1 -- 172.21.15.7:0/3622774235 >> v1:172.21.15.83:6789/0 conn(0x7f346c13bac0 legacy=0x7f346c071390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.630+0000 7f346bfff640 10 monclient: _finish_hunting 0 2024-03-28T16:15:50.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.630+0000 7f346bfff640 1 monclient: found mon.noname-b 2024-03-28T16:15:50.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.630+0000 7f346bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:50.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.630+0000 7f346bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-03-28T16:15:50.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.630+0000 7f346bfff640 1 -- 172.21.15.7:0/3622774235 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346c071aa0 con 0x7f346c06f780 2024-03-28T16:15:50.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.631+0000 7fd3122b0640 10 monclient(hunting): handle_auth_done global_id 4309 payload 274 2024-03-28T16:15:50.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.631+0000 7fd3122b0640 1 -- 172.21.15.7:0/3390582447 >> v2:172.21.15.7:3300/0 conn(0x7fd30c0ab4b0 msgr2=0x7fd30c149440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:50.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.631+0000 7fd3122b0640 1 --2- 172.21.15.7:0/3390582447 >> v2:172.21.15.7:3300/0 conn(0x7fd30c0ab4b0 0x7fd30c149440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.631+0000 7fd3122b0640 1 -- 172.21.15.7:0/3390582447 >> v1:172.21.15.7:6789/0 conn(0x7fd30c0a9dc0 legacy=0x7fd30c142cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:50.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.631+0000 7fd3122b0640 10 monclient: _finish_hunting 0 2024-03-28T16:15:50.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.631+0000 7fd3122b0640 1 monclient: found mon.noname-c 2024-03-28T16:15:50.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.631+0000 7fd3122b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:50.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.631+0000 7fd3122b0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3300/0 2024-03-28T16:15:50.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.631+0000 7fd3122b0640 1 -- 172.21.15.7:0/3390582447 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30c148f20 con 0x7fd30c0a8b60 2024-03-28T16:15:50.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.632+0000 7f5fa5ec5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.632+0000 7f5fa5ec5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.632+0000 7f5fa5ec5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:50.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.633+0000 7f5fa56c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.633+0000 7f5fa56c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.633+0000 7f5fa56c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:50.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.633+0000 7f346bfff640 1 --2- 172.21.15.7:0/3622774235 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f346c06f780 0x7f346c070e50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3464009e50 tx=0x7f346402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f576332c706278b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.634+0000 7f5fa5ec5640 10 monclient(hunting): handle_auth_done global_id 4315 payload 274 2024-03-28T16:15:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.634+0000 7f5fa5ec5640 1 -- 172.21.15.7:0/3513372345 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5fa0071810 msgr2=0x7f5fa0149000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.634+0000 7f5fa5ec5640 1 --2- 172.21.15.7:0/3513372345 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5fa0071810 0x7f5fa0149000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.634+0000 7f5fa5ec5640 1 -- 172.21.15.7:0/3513372345 >> v1:172.21.15.83:6789/0 conn(0x7f5fa0149540 legacy=0x7f5fa0145790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.634+0000 7f5fa5ec5640 10 monclient: _finish_hunting 0 2024-03-28T16:15:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.634+0000 7f5fa5ec5640 1 monclient: found mon.noname-c 2024-03-28T16:15:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.634+0000 7f5fa5ec5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.634+0000 7f5fa5ec5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3300/0 2024-03-28T16:15:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.634+0000 7f5fa5ec5640 1 -- 172.21.15.7:0/3513372345 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa0069cc0 con 0x7f5fa006a490 2024-03-28T16:15:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.635+0000 7fd3122b0640 1 --2- 172.21.15.7:0/3390582447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd30c0a8b60 0x7fd30c0a8f40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd30c06bbe0 tx=0x7fd30402be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4feb6986110d6b4f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.637+0000 7f0753fff640 1 --2- 172.21.15.7:0/2172790457 >> v2:172.21.15.7:3300/0 conn(0x7f075413e000 0x7f075413d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.637+0000 7f7a9285c640 10 monclient: get_monmap_and_config 2024-03-28T16:15:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.637+0000 7f7a9285c640 10 monclient: build_initial_monmap 2024-03-28T16:15:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.637+0000 7f7a9285c640 10 monclient: monmap: 2024-03-28T16:15:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:50.639432+0000 2024-03-28T16:15:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:50.639432+0000 2024-03-28T16:15:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:50.663 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:50.663 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:50.663 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:50.663 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.638+0000 7f51cd407640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f51c813eab0 0x7f51c81412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.638+0000 7f7a9285c640 1 Processor -- start 2024-03-28T16:15:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.638+0000 7f7a9285c640 1 -- start start 2024-03-28T16:15:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.638+0000 7f7a9285c640 10 monclient: init 2024-03-28T16:15:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.639+0000 7febbbfff640 1 -- 172.21.15.7:0/2217813657 <== mon.1 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317705361 0 0) 0x7febc40aa420 con 0x7febc41408d0 2024-03-28T16:15:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.639+0000 7febbbfff640 10 monclient(hunting): _negotiate 2024-03-28T16:15:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.639+0000 7febbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.639+0000 7febbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.639+0000 7febbbfff640 10 monclient(hunting): my global_id is 4318 2024-03-28T16:15:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.639+0000 7febbbfff640 1 -- 172.21.15.7:0/2217813657 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febb0003520 con 0x7febc41408d0 2024-03-28T16:15:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.639+0000 7f7a9285c640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.639+0000 7f7a9285c640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-03-28T16:15:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.639+0000 7f7a9285c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7a8c071920 0x7f7a8c13e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.639+0000 7f7a9285c640 10 monclient(hunting): picked mon.noname-a con 0x7f7a8c071920 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:50.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.639+0000 7f7a9285c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a8c0a89a0 0x7f7a8c13ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.639+0000 7f7a9285c640 10 monclient(hunting): picked mon.noname-b con 0x7f7a8c0a89a0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:50.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.639+0000 7f7a9285c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a8c0b1890 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:50.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.639+0000 7f7a9285c640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8c0a8ec0 con 0x7f7a8c0b1890 2024-03-28T16:15:50.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.639+0000 7f7a9285c640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.639+0000 7f7a9285c640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a8c0aa480 con 0x7f7a8c071920 2024-03-28T16:15:50.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.639+0000 7f7a9285c640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.639+0000 7f7a9285c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c0b2430 con 0x7f7a8c0a89a0 2024-03-28T16:15:50.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.639+0000 7f7a9285c640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:50.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.639+0000 7f7a9285c640 10 monclient(hunting): authenticate will time out at 1417.781414s 2024-03-28T16:15:50.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.640+0000 7f0753fff640 10 monclient(hunting): get_auth_request con 0x7f075413e000 auth_method 0 2024-03-28T16:15:50.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.640+0000 7f0753fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.640+0000 7f0753fff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.640+0000 7f0753fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.641+0000 7f51cd407640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f51c813eab0 0x7f51c81412d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60132/0 (socket says 172.21.15.7:60132) 2024-03-28T16:15:50.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.641+0000 7f51cd407640 1 -- 172.21.15.7:0/2611555702 learned_addr learned my addr 172.21.15.7:0/2611555702 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:50.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.641+0000 7f51cd407640 10 monclient(hunting): get_auth_request con 0x7f51c813eab0 auth_method 0 2024-03-28T16:15:50.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.641+0000 7f51cd407640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.641+0000 7f51cd407640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.641+0000 7f51cd407640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.641+0000 7f7a8bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7a8c071920 0x7f7a8c13e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.641+0000 7f0753fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.641+0000 7f0753fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.641+0000 7f0753fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:50.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f346b7fe640 1 -- 172.21.15.7:0/3622774235 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346402ca70 con 0x7f346c06f780 2024-03-28T16:15:50.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f346b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f346b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:50.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f346b7fe640 10 monclient: dump: 2024-03-28T16:15:50.672 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.672 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.672 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.672 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.672 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.672 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.672 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.672 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.672 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.672 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f346b7fe640 10 monclient: _finish_auth 0 2024-03-28T16:15:50.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f346b7fe640 10 monclient: _check_auth_tickets 2024-03-28T16:15:50.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f346b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:50.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f346b7fe640 1 -- 172.21.15.7:0/3622774235 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346402cbd0 con 0x7f346c06f780 2024-03-28T16:15:50.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f3473173640 5 monclient: authenticate success, global_id 4268 2024-03-28T16:15:50.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f346b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:50.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f346b7fe640 1 -- 172.21.15.7:0/3622774235 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34640343f0 con 0x7f346c06f780 2024-03-28T16:15:50.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f346b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f346b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:15:50.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f346b7fe640 10 monclient: dump: 2024-03-28T16:15:50.673 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.673 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.674 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.674 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.674 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.674 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.674 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.674 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.674 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.674 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f3473173640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:50.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f3473173640 4 set_mon_vals no callback set 2024-03-28T16:15:50.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f51cd407640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f51cd407640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f51cd407640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:50.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f3473173640 10 monclient: shutdown 2024-03-28T16:15:50.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f3473173640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:50.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f3473173640 1 -- 172.21.15.7:0/3622774235 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f346c06f780 msgr2=0x7f346c070e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f3473173640 1 --2- 172.21.15.7:0/3622774235 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f346c06f780 0x7f346c070e50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3464009e50 tx=0x7f346402bed0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f3473173640 1 -- 172.21.15.7:0/3622774235 shutdown_connections 2024-03-28T16:15:50.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f3473173640 1 --2- 172.21.15.7:0/3622774235 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f346c06f780 0x7f346c070e50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.642+0000 7f3473173640 1 --2- 172.21.15.7:0/3622774235 >> v2:172.21.15.83:3300/0 conn(0x7f346c0ad840 0x7f346c13b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.643+0000 7f3473173640 1 -- 172.21.15.7:0/3622774235 >> 172.21.15.7:0/3622774235 conn(0x7f346c0b2db0 msgr2=0x7f346c0b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:50.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.643+0000 7f3473173640 1 -- 172.21.15.7:0/3622774235 shutdown_connections 2024-03-28T16:15:50.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.643+0000 7f3473173640 1 -- 172.21.15.7:0/3622774235 wait complete. 2024-03-28T16:15:50.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.643+0000 7f0753fff640 10 monclient(hunting): handle_auth_done global_id 4308 payload 274 2024-03-28T16:15:50.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.643+0000 7f3473173640 10 monclient: build_initial_monmap 2024-03-28T16:15:50.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.643+0000 7f3473173640 10 monclient: monmap: 2024-03-28T16:15:50.677 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:50.677 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:50.677 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:50.677 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:50.677 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:50.677 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.678 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:50.678 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:50.678 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:50.678 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.643+0000 7f0753fff640 1 -- 172.21.15.7:0/2172790457 >> v1:172.21.15.83:6789/0 conn(0x7f075413d3e0 legacy=0x7f075413a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.643+0000 7f0753fff640 1 -- 172.21.15.7:0/2172790457 >> v1:172.21.15.7:6789/0 conn(0x7f07540ada00 legacy=0x7f0754147c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:50.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.643+0000 7f0753fff640 10 monclient: _finish_hunting 0 2024-03-28T16:15:50.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.643+0000 7f0753fff640 1 monclient: found mon.noname-a 2024-03-28T16:15:50.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.643+0000 7f0753fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:50.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.643+0000 7f0753fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:15:50.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.643+0000 7f0753fff640 1 -- 172.21.15.7:0/2172790457 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07540aa210 con 0x7f075413e000 2024-03-28T16:15:50.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.643+0000 7f3473173640 1 librados: starting msgr at 2024-03-28T16:15:50.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.643+0000 7f3473173640 1 librados: starting objecter 2024-03-28T16:15:50.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.643+0000 7f51ccc06640 1 --2- 172.21.15.7:0/2611555702 >> v2:172.21.15.83:3300/0 conn(0x7f51c80b3b90 0x7f51c80a86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.644+0000 7f7a8bfff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7a8c071920 0x7f7a8c13e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37828/0 (socket says 172.21.15.7:37828) 2024-03-28T16:15:50.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.644+0000 7f7a8bfff640 1 -- 172.21.15.7:0/2692486868 learned_addr learned my addr 172.21.15.7:0/2692486868 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:50.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.644+0000 7f7a8bfff640 10 monclient(hunting): get_auth_request con 0x7f7a8c071920 auth_method 0 2024-03-28T16:15:50.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.644+0000 7f7a8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.644+0000 7f7a8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.644+0000 7f7a8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.644+0000 7f3473173640 1 Processor -- start 2024-03-28T16:15:50.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.644+0000 7f3473173640 1 -- start start 2024-03-28T16:15:50.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.644+0000 7f3473173640 1 librados: setting wanted keys 2024-03-28T16:15:50.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.644+0000 7f3473173640 1 librados: calling monclient init 2024-03-28T16:15:50.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.644+0000 7f3473173640 10 monclient: init 2024-03-28T16:15:50.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.644+0000 7f3473173640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:50.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.644+0000 7f3473173640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-28T16:15:50.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.644+0000 7f3473173640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f346c06f780 0x7f346c0b07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.644+0000 7f3473173640 10 monclient(hunting): picked mon.noname-b con 0x7f346c06f780 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:50.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.644+0000 7f3473173640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f346c0ad840 0x7f346c0b2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.644+0000 7f51cd407640 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2024-03-28T16:15:50.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.645+0000 7f3473173640 10 monclient(hunting): picked mon.noname-c con 0x7f346c0ad840 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:50.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.645+0000 7f3473173640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f346c13bac0 0x7f346c12c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.645+0000 7f3473173640 10 monclient(hunting): picked mon.noname-a con 0x7f346c13bac0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:50.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.645+0000 7f3473173640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.645+0000 7f3473173640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346c0b0d00 con 0x7f346c13bac0 2024-03-28T16:15:50.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.645+0000 7f3473173640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.645+0000 7f3473173640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346c0ab8c0 con 0x7f346c06f780 2024-03-28T16:15:50.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.645+0000 7f3473173640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.645+0000 7f3473173640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346c13d8e0 con 0x7f346c0ad840 2024-03-28T16:15:50.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.645+0000 7f3473173640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:50.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.645+0000 7f3473173640 10 monclient(hunting): authenticate will time out at 1417.786917s 2024-03-28T16:15:50.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.645+0000 7f51cd407640 1 -- 172.21.15.7:0/2611555702 >> v2:172.21.15.83:3300/0 conn(0x7f51c80b3b90 msgr2=0x7f51c80a86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:50.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.645+0000 7f51cd407640 1 --2- 172.21.15.7:0/2611555702 >> v2:172.21.15.83:3300/0 conn(0x7f51c80b3b90 0x7f51c80a86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.645+0000 7f51cd407640 1 -- 172.21.15.7:0/2611555702 >> v2:172.21.15.7:3300/0 conn(0x7f51c80b32e0 msgr2=0x7f51c813e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:50.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.645+0000 7f51cd407640 1 --2- 172.21.15.7:0/2611555702 >> v2:172.21.15.7:3300/0 conn(0x7f51c80b32e0 0x7f51c813e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.645+0000 7f51cd407640 10 monclient: _finish_hunting 0 2024-03-28T16:15:50.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.645+0000 7f51cd407640 1 monclient: found mon.noname-b 2024-03-28T16:15:50.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.645+0000 7f51cd407640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:50.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.645+0000 7f51cd407640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-03-28T16:15:50.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.645+0000 7f51cd407640 1 -- 172.21.15.7:0/2611555702 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c80a92d0 con 0x7f51c813eab0 2024-03-28T16:15:50.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 1 -- 172.21.15.7:0/2217813657 <== mon.1 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 983648555 0 0) 0x7febb0003520 con 0x7febc41408d0 2024-03-28T16:15:50.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 1 -- 172.21.15.7:0/2217813657 >> v2:172.21.15.7:3300/0 conn(0x7febc4140cb0 msgr2=0x7febc4141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:50.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 1 --2- 172.21.15.7:0/2217813657 >> v2:172.21.15.7:3300/0 conn(0x7febc4140cb0 0x7febc4141380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 1 -- 172.21.15.7:0/2217813657 >> v1:172.21.15.7:6789/0 conn(0x7febc40ab950 legacy=0x7febc4147c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:50.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 10 monclient: _finish_hunting 0 2024-03-28T16:15:50.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 1 monclient: found mon.noname-c-legacy 2024-03-28T16:15:50.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:50.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.83:6789/0 2024-03-28T16:15:50.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 1 -- 172.21.15.7:0/2217813657 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc40a8d90 con 0x7febc41408d0 2024-03-28T16:15:50.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 10 monclient: _finish_auth 0 2024-03-28T16:15:50.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 10 monclient: _check_auth_tickets 2024-03-28T16:15:50.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:50.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 1 -- 172.21.15.7:0/2217813657 <== mon.1 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1105724391 0 0) 0x7febc0001670 con 0x7febc41408d0 2024-03-28T16:15:50.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7f7a8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7f7a8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7f7a8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:50.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-03-28T16:15:50.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 10 monclient: dump: 2024-03-28T16:15:50.687 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.687 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.687 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.687 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.687 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.687 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.687 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.687 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.687 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.687 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] but i'm connected to v1:172.21.15.83:6789/0, reconnecting 2024-03-28T16:15:50.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:50.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 1 -- 172.21.15.7:0/2217813657 >> v1:172.21.15.83:6789/0 conn(0x7febc41408d0 legacy=0x7febc413a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-28T16:15:50.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 1 --2- 172.21.15.7:0/2217813657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febb0003970 0x7febb0003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 10 monclient(hunting): picked mon.b con 0x7febb0003970 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:50.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 1 --2- 172.21.15.7:0/2217813657 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7febb00047c0 0x7febb0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 10 monclient(hunting): picked mon.c con 0x7febb00047c0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:50.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 1 --2- 172.21.15.7:0/2217813657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febb00052a0 0x7febb0005950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 10 monclient(hunting): picked mon.a con 0x7febb00052a0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:50.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 1 -- 172.21.15.7:0/2217813657 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc40ab250 con 0x7febb00052a0 2024-03-28T16:15:50.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 1 -- 172.21.15.7:0/2217813657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb0006230 con 0x7febb0003970 2024-03-28T16:15:50.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 1 -- 172.21.15.7:0/2217813657 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb00063f0 con 0x7febb00047c0 2024-03-28T16:15:50.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:50.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.646+0000 7febbbfff640 10 monclient(hunting): _finish_auth 0 2024-03-28T16:15:50.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.647+0000 7f0753fff640 1 --2- 172.21.15.7:0/2172790457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f075413e000 0x7f075413d7c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0744007200 tx=0x7f074402da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d6fa31fa2c0f449 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.649+0000 7f51cd407640 1 --2- 172.21.15.7:0/2611555702 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f51c813eab0 0x7f51c81412d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f51c80b4240 tx=0x7f51c002bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7562200524c19118 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.649+0000 7f7a8bfff640 10 monclient(hunting): handle_auth_done global_id 4311 payload 274 2024-03-28T16:15:50.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.649+0000 7f7a8bfff640 1 -- 172.21.15.7:0/2692486868 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a8c0a89a0 msgr2=0x7f7a8c13ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:50.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.649+0000 7f7a8bfff640 1 --2- 172.21.15.7:0/2692486868 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a8c0a89a0 0x7f7a8c13ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.649+0000 7f7a8bfff640 1 -- 172.21.15.7:0/2692486868 >> v1:172.21.15.7:6789/0 conn(0x7f7a8c0b1890 legacy=0x7f7a8c0b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:50.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.649+0000 7f7a8bfff640 10 monclient: _finish_hunting 0 2024-03-28T16:15:50.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.649+0000 7f7a8bfff640 1 monclient: found mon.noname-a 2024-03-28T16:15:50.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.649+0000 7f7a8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:50.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.649+0000 7f7a8bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:15:50.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.649+0000 7f7a8bfff640 1 -- 172.21.15.7:0/2692486868 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8c0aa660 con 0x7f7a8c071920 2024-03-28T16:15:50.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.651+0000 7f5fa5ec5640 1 --2- 172.21.15.7:0/3513372345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5fa006a490 0x7f5fa00712d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c0097d0 tx=0x7f5f9c02b780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5cbcd821716595e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.653+0000 7f346bfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f346c0ad840 0x7f346c0b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.654+0000 7f346bfff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f346c0ad840 0x7f346c0b2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60162/0 (socket says 172.21.15.7:60162) 2024-03-28T16:15:50.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.654+0000 7f346bfff640 1 -- 172.21.15.7:0/3791127862 learned_addr learned my addr 172.21.15.7:0/3791127862 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:50.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.654+0000 7f346bfff640 10 monclient(hunting): get_auth_request con 0x7f346c0ad840 auth_method 0 2024-03-28T16:15:50.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.654+0000 7f346bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.654+0000 7f346bfff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.654+0000 7f346bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.654+0000 7f346bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.654+0000 7f346bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.654+0000 7f346bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:50.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.657+0000 7f346bfff640 10 monclient(hunting): handle_auth_done global_id 4277 payload 824 2024-03-28T16:15:50.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.657+0000 7f346bfff640 1 -- 172.21.15.7:0/3791127862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f346c06f780 msgr2=0x7f346c0b07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.657+0000 7f346bfff640 1 --2- 172.21.15.7:0/3791127862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f346c06f780 0x7f346c0b07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.657+0000 7f346bfff640 1 -- 172.21.15.7:0/3791127862 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f346c13bac0 msgr2=0x7f346c12c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.657+0000 7f346bfff640 1 --2- 172.21.15.7:0/3791127862 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f346c13bac0 0x7f346c12c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.657+0000 7f346bfff640 10 monclient: _finish_hunting 0 2024-03-28T16:15:50.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.657+0000 7f346bfff640 1 monclient: found mon.noname-c 2024-03-28T16:15:50.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.657+0000 7f346bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:50.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.657+0000 7f346bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-03-28T16:15:50.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.657+0000 7f346bfff640 1 -- 172.21.15.7:0/3791127862 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3464009b30 con 0x7f346c0ad840 2024-03-28T16:15:50.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.658+0000 7f6c0d2d4640 1 --2- 172.21.15.7:0/3234525566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c00032eb0 0x7f6c000cc060 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4009870 tx=0x7f6bf402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5800fb4df11a538 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.659+0000 7fd3112ae640 1 -- 172.21.15.7:0/3390582447 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30402c9f0 con 0x7fd30c0a8b60 2024-03-28T16:15:50.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.659+0000 7fd3112ae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.659+0000 7fd3112ae640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:50.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.659+0000 7fd3112ae640 10 monclient: dump: 2024-03-28T16:15:50.694 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.695 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.695 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.695 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.695 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.695 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.695 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.695 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.695 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.695 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.659+0000 7fd3112ae640 10 monclient: _finish_auth 0 2024-03-28T16:15:50.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.659+0000 7fd3112ae640 10 monclient: _check_auth_tickets 2024-03-28T16:15:50.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.659+0000 7fd3112ae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:50.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.659+0000 7fd3112ae640 1 -- 172.21.15.7:0/3390582447 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30402cb50 con 0x7fd30c0a8b60 2024-03-28T16:15:50.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.659+0000 7fd314514640 5 monclient: authenticate success, global_id 4309 2024-03-28T16:15:50.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.659+0000 7fd314514640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:50.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.659+0000 7fd3112ae640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:50.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.659+0000 7fd314514640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:50.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.659+0000 7fd314514640 4 set_mon_vals no callback set 2024-03-28T16:15:50.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.659+0000 7fd3112ae640 1 -- 172.21.15.7:0/3390582447 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3040343f0 con 0x7fd30c0a8b60 2024-03-28T16:15:50.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.659+0000 7fd3112ae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.659+0000 7fd3112ae640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:50.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.659+0000 7fd3112ae640 10 monclient: dump: 2024-03-28T16:15:50.696 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.696 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.696 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.696 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.696 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.696 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.696 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.697 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.697 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.697 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7f51c77fe640 1 -- 172.21.15.7:0/2611555702 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c002c950 con 0x7f51c813eab0 2024-03-28T16:15:50.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7f51c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7f51c77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:50.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7f51c77fe640 10 monclient: dump: 2024-03-28T16:15:50.699 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.699 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.699 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.699 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.699 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.699 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.699 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.699 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.699 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.700 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7f51c77fe640 10 monclient: _finish_auth 0 2024-03-28T16:15:50.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7f51c77fe640 10 monclient: _check_auth_tickets 2024-03-28T16:15:50.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7f51c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:50.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7f51c77fe640 1 -- 172.21.15.7:0/2611555702 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c002cab0 con 0x7f51c813eab0 2024-03-28T16:15:50.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7f51c77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:50.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7f51c77fe640 1 -- 172.21.15.7:0/2611555702 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c002ce40 con 0x7f51c813eab0 2024-03-28T16:15:50.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7f51c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7f51c77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:15:50.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7f51c77fe640 10 monclient: dump: 2024-03-28T16:15:50.700 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.700 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.700 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.700 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.700 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.701 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.701 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.701 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.701 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.701 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7f51cdc08640 5 monclient: authenticate success, global_id 4274 2024-03-28T16:15:50.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7febc8e1f640 1 --2- 172.21.15.7:0/2217813657 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7febb00047c0 0x7febb0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7f51cdc08640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:50.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7f51cdc08640 4 set_mon_vals no callback set 2024-03-28T16:15:50.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7fd314514640 10 monclient: shutdown 2024-03-28T16:15:50.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7fd314514640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:50.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7fd314514640 1 -- 172.21.15.7:0/3390582447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd30c0a8b60 msgr2=0x7fd30c0a8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7fd314514640 1 --2- 172.21.15.7:0/3390582447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd30c0a8b60 0x7fd30c0a8f40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd30c06bbe0 tx=0x7fd30402be50 comp rx=0 tx=0).stop 2024-03-28T16:15:50.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7fd314514640 1 -- 172.21.15.7:0/3390582447 shutdown_connections 2024-03-28T16:15:50.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7f51cdc08640 10 monclient: shutdown 2024-03-28T16:15:50.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.661+0000 7f51cdc08640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:50.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.661+0000 7f51cdc08640 1 -- 172.21.15.7:0/2611555702 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f51c813eab0 msgr2=0x7f51c81412d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.661+0000 7f51cdc08640 1 --2- 172.21.15.7:0/2611555702 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f51c813eab0 0x7f51c81412d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f51c80b4240 tx=0x7f51c002bdb0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7fd314514640 1 --2- 172.21.15.7:0/3390582447 >> v2:172.21.15.7:3300/0 conn(0x7fd30c0ab4b0 0x7fd30c149440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7fd314514640 1 --2- 172.21.15.7:0/3390582447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd30c0a8b60 0x7fd30c0a8f40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7fd314514640 1 -- 172.21.15.7:0/3390582447 >> 172.21.15.7:0/3390582447 conn(0x7fd30c0b2cb0 msgr2=0x7fd30c0b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:50.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7fd314514640 1 -- 172.21.15.7:0/3390582447 shutdown_connections 2024-03-28T16:15:50.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.660+0000 7fd314514640 1 -- 172.21.15.7:0/3390582447 wait complete. 2024-03-28T16:15:50.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.661+0000 7f51cdc08640 1 -- 172.21.15.7:0/2611555702 shutdown_connections 2024-03-28T16:15:50.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.661+0000 7f51cdc08640 1 --2- 172.21.15.7:0/2611555702 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f51c813eab0 0x7f51c81412d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.661+0000 7f51cdc08640 1 --2- 172.21.15.7:0/2611555702 >> v2:172.21.15.7:3300/0 conn(0x7f51c80b32e0 0x7f51c813e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.661+0000 7f51cdc08640 1 --2- 172.21.15.7:0/2611555702 >> v2:172.21.15.83:3300/0 conn(0x7f51c80b3b90 0x7f51c80a86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.661+0000 7f51cdc08640 1 -- 172.21.15.7:0/2611555702 >> 172.21.15.7:0/2611555702 conn(0x7f51c80711a0 msgr2=0x7f51c80715b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:50.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.661+0000 7f51cdc08640 1 -- 172.21.15.7:0/2611555702 shutdown_connections 2024-03-28T16:15:50.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.661+0000 7f51cdc08640 1 -- 172.21.15.7:0/2611555702 wait complete. 2024-03-28T16:15:50.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.661+0000 7f7a8bfff640 1 --2- 172.21.15.7:0/2692486868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a8c071920 0x7f7a8c13e830 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c009870 tx=0x7f7a7c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f1427a04c1af76d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.661+0000 7fd314514640 10 monclient: build_initial_monmap 2024-03-28T16:15:50.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.661+0000 7fd314514640 10 monclient: monmap: 2024-03-28T16:15:50.707 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:50.707 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:50.707 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:50.707 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:50.707 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:50.707 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.707 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:50.707 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:50.707 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:50.707 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7febc8e1f640 10 monclient(hunting): get_auth_request con 0x7febb00047c0 auth_method 0 2024-03-28T16:15:50.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7febc8e1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7febc8e1f640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7febc8e1f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:15:50.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 1 librados: starting msgr at 2024-03-28T16:15:50.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 1 librados: starting objecter 2024-03-28T16:15:50.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 1 Processor -- start 2024-03-28T16:15:50.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7f51cdc08640 10 monclient: build_initial_monmap 2024-03-28T16:15:50.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7f51cdc08640 10 monclient: monmap: 2024-03-28T16:15:50.709 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:50.709 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:50.709 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:50.710 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:50.710 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:50.710 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.710 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:50.710 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:50.710 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:50.710 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 1 -- start start 2024-03-28T16:15:50.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 1 librados: setting wanted keys 2024-03-28T16:15:50.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 1 librados: calling monclient init 2024-03-28T16:15:50.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 10 monclient: init 2024-03-28T16:15:50.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7f346bfff640 1 --2- 172.21.15.7:0/3791127862 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f346c0ad840 0x7f346c0b2400 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3464009e50 tx=0x7f3464004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7f34697fa640 10 client.?.objecter ms_handle_connect 0x7f346c0ad840 2024-03-28T16:15:50.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7f34697fa640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:50.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7f51cdc08640 1 librados: starting msgr at 2024-03-28T16:15:50.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7f51cdc08640 1 librados: starting objecter 2024-03-28T16:15:50.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:50.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-28T16:15:50.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd30c0a8b60 0x7fd30c12d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7f51cdc08640 1 Processor -- start 2024-03-28T16:15:50.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 10 monclient(hunting): picked mon.noname-b con 0x7fd30c0a8b60 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:50.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd30c0a9dc0 0x7fd30c12dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 10 monclient(hunting): picked mon.noname-c con 0x7fd30c0a9dc0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:50.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd30c0ab4b0 0x7fd30c126190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 10 monclient(hunting): picked mon.noname-a con 0x7fd30c0ab4b0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:50.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30c12e400 con 0x7fd30c0ab4b0 2024-03-28T16:15:50.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30c13ee70 con 0x7fd30c0a8b60 2024-03-28T16:15:50.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd30c1296d0 con 0x7fd30c0a9dc0 2024-03-28T16:15:50.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:50.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.662+0000 7fd314514640 10 monclient(hunting): authenticate will time out at 1417.804712s 2024-03-28T16:15:50.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f51cdc08640 1 -- start start 2024-03-28T16:15:50.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f5fa4ec3640 1 -- 172.21.15.7:0/3513372345 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c030710 con 0x7f5fa006a490 2024-03-28T16:15:50.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f5fa4ec3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f5fa4ec3640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:50.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f5fa4ec3640 10 monclient: dump: 2024-03-28T16:15:50.715 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.716 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.716 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.716 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.716 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.716 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.716 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.716 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.716 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.716 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f5fa4ec3640 10 monclient: _finish_auth 0 2024-03-28T16:15:50.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f5fa4ec3640 10 monclient: _check_auth_tickets 2024-03-28T16:15:50.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f5fa4ec3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:50.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f5fa4ec3640 1 -- 172.21.15.7:0/3513372345 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9c02e050 con 0x7f5fa006a490 2024-03-28T16:15:50.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f5fa6ec7640 5 monclient: authenticate success, global_id 4315 2024-03-28T16:15:50.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f5fa6ec7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:50.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f5fa4ec3640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:50.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f5fa4ec3640 1 -- 172.21.15.7:0/3513372345 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c030a90 con 0x7f5fa006a490 2024-03-28T16:15:50.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f5fa6ec7640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:50.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f5fa4ec3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f5fa6ec7640 4 set_mon_vals no callback set 2024-03-28T16:15:50.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f5fa4ec3640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:50.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f5fa4ec3640 10 monclient: dump: 2024-03-28T16:15:50.717 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.717 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.717 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.717 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.717 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.717 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.718 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.718 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.718 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.718 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f51cdc08640 1 librados: setting wanted keys 2024-03-28T16:15:50.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f51cdc08640 1 librados: calling monclient init 2024-03-28T16:15:50.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f51cdc08640 10 monclient: init 2024-03-28T16:15:50.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7febc9620640 1 --2- 172.21.15.7:0/2217813657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febb0003970 0x7febb0003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f5fa6ec7640 10 monclient: shutdown 2024-03-28T16:15:50.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f5fa6ec7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:50.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f5fa6ec7640 1 -- 172.21.15.7:0/3513372345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5fa006a490 msgr2=0x7f5fa00712d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f5fa6ec7640 1 --2- 172.21.15.7:0/3513372345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5fa006a490 0x7f5fa00712d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c0097d0 tx=0x7f5f9c02b780 comp rx=0 tx=0).stop 2024-03-28T16:15:50.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.663+0000 7f51cdc08640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:50.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.664+0000 7f5fa6ec7640 1 -- 172.21.15.7:0/3513372345 shutdown_connections 2024-03-28T16:15:50.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.664+0000 7f5fa6ec7640 1 --2- 172.21.15.7:0/3513372345 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5fa0071810 0x7f5fa0149000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.664+0000 7f5fa6ec7640 1 --2- 172.21.15.7:0/3513372345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5fa006a490 0x7f5fa00712d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.664+0000 7f5fa6ec7640 1 -- 172.21.15.7:0/3513372345 >> 172.21.15.7:0/3513372345 conn(0x7f5fa00b1ae0 msgr2=0x7f5fa00b1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:50.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.664+0000 7f51cdc08640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-28T16:15:50.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.664+0000 7f51cdc08640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51c80b32e0 0x7f51c8143710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.664+0000 7f51cdc08640 10 monclient(hunting): picked mon.noname-b con 0x7f51c80b32e0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:50.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.664+0000 7f51cdc08640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51c80b3b90 0x7f51c80acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.664+0000 7f51cdc08640 10 monclient(hunting): picked mon.noname-a con 0x7f51c80b3b90 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:50.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.664+0000 7f51cdc08640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f51c813eab0 0x7f51c80aba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.664+0000 7f51cdc08640 10 monclient(hunting): picked mon.noname-c con 0x7f51c813eab0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:50.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.664+0000 7f51cdc08640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.664+0000 7f51cdc08640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c80ad040 con 0x7f51c80b3b90 2024-03-28T16:15:50.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.664+0000 7f51cdc08640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.664+0000 7f51cdc08640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c80ac5e0 con 0x7f51c80b32e0 2024-03-28T16:15:50.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.664+0000 7f51cdc08640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.664+0000 7f51cdc08640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c80abf80 con 0x7f51c813eab0 2024-03-28T16:15:50.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.664+0000 7f51cdc08640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:50.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.664+0000 7f51cdc08640 10 monclient(hunting): authenticate will time out at 1417.805923s 2024-03-28T16:15:50.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.664+0000 7f5fa6ec7640 1 -- 172.21.15.7:0/3513372345 shutdown_connections 2024-03-28T16:15:50.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.664+0000 7f5fa6ec7640 1 -- 172.21.15.7:0/3513372345 wait complete. 2024-03-28T16:15:50.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.665+0000 7febc9620640 10 monclient(hunting): get_auth_request con 0x7febb0003970 auth_method 0 2024-03-28T16:15:50.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.665+0000 7febc9620640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.665+0000 7febc9620640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.665+0000 7febc9620640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:15:50.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.665+0000 7f5fa6ec7640 10 monclient: build_initial_monmap 2024-03-28T16:15:50.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.665+0000 7f5fa6ec7640 10 monclient: monmap: 2024-03-28T16:15:50.724 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:50.724 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:50.724 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:50.724 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:50.724 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:50.724 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.724 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:50.724 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:50.724 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:50.724 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.665+0000 7f5fa6ec7640 1 librados: starting msgr at 2024-03-28T16:15:50.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.665+0000 7f5fa6ec7640 1 librados: starting objecter 2024-03-28T16:15:50.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.665+0000 7f5fa6ec7640 1 Processor -- start 2024-03-28T16:15:50.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.665+0000 7f5fa6ec7640 1 -- start start 2024-03-28T16:15:50.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.665+0000 7f5fa6ec7640 1 librados: setting wanted keys 2024-03-28T16:15:50.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.665+0000 7f5fa6ec7640 1 librados: calling monclient init 2024-03-28T16:15:50.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.665+0000 7f5fa6ec7640 10 monclient: init 2024-03-28T16:15:50.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.666+0000 7febc9620640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.666+0000 7febc9620640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.666+0000 7febc9620640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-28T16:15:50.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.666+0000 7febc9e21640 1 --2- 172.21.15.7:0/2217813657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febb00052a0 0x7febb0005950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.666+0000 7f5fa6ec7640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:50.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.666+0000 7f5fa6ec7640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-28T16:15:50.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.666+0000 7f5fa6ec7640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5fa006a490 0x7f5fa0144400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.666+0000 7f5fa6ec7640 10 monclient(hunting): picked mon.noname-c con 0x7f5fa006a490 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:50.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.666+0000 7f5fa6ec7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5fa0071810 0x7f5fa0144a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.666+0000 7f5fa6ec7640 10 monclient(hunting): picked mon.noname-b con 0x7f5fa0071810 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:50.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.666+0000 7f5fa6ec7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fa0149540 0x7f5fa0140700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.666+0000 7f5fa6ec7640 10 monclient(hunting): picked mon.noname-a con 0x7f5fa0149540 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:50.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f5fa6ec7640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7fd3122b0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd30c0a8b60 0x7fd30c12d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f07537fe640 1 -- 172.21.15.7:0/2172790457 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0744008df0 con 0x7f075413e000 2024-03-28T16:15:50.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f07537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f07537fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:15:50.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f5fa6ec7640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa0146510 con 0x7f5fa0149540 2024-03-28T16:15:50.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f07537fe640 10 monclient: dump: 2024-03-28T16:15:50.728 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.728 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.728 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.729 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.729 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.729 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.729 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.729 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.729 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.729 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f07537fe640 10 monclient: _finish_auth 0 2024-03-28T16:15:50.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f07537fe640 10 monclient: _check_auth_tickets 2024-03-28T16:15:50.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f07537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:50.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f07537fe640 1 -- 172.21.15.7:0/2172790457 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f074400a0d0 con 0x7f075413e000 2024-03-28T16:15:50.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f07537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:50.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f5fa6ec7640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f07537fe640 1 -- 172.21.15.7:0/2172790457 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074400a460 con 0x7f075413e000 2024-03-28T16:15:50.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f07537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f07537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:15:50.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f07537fe640 10 monclient: dump: 2024-03-28T16:15:50.730 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.730 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.730 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.730 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.730 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.730 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.730 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.731 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.731 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.731 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f075b037640 5 monclient: authenticate success, global_id 4308 2024-03-28T16:15:50.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f075b037640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:50.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f5fa6ec7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa0145160 con 0x7f5fa0071810 2024-03-28T16:15:50.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f075b037640 4 set_mon_vals no callback set 2024-03-28T16:15:50.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f5fa6ec7640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f5fa6ec7640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa0143590 con 0x7f5fa006a490 2024-03-28T16:15:50.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f5fa6ec7640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:50.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f5fa6ec7640 10 monclient(hunting): authenticate will time out at 1417.809105s 2024-03-28T16:15:50.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7febc8e1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7febc8e1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7febc8e1f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-28T16:15:50.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7febc9e21640 10 monclient(hunting): get_auth_request con 0x7febb00052a0 auth_method 0 2024-03-28T16:15:50.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7febc9e21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7febc9e21640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7febc9e21640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:15:50.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f075b037640 10 monclient: shutdown 2024-03-28T16:15:50.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f075b037640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:50.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f075b037640 1 -- 172.21.15.7:0/2172790457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f075413e000 msgr2=0x7f075413d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.667+0000 7f075b037640 1 --2- 172.21.15.7:0/2172790457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f075413e000 0x7f075413d7c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0744007200 tx=0x7f074402da50 comp rx=0 tx=0).stop 2024-03-28T16:15:50.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7f075b037640 1 -- 172.21.15.7:0/2172790457 shutdown_connections 2024-03-28T16:15:50.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7f075b037640 1 --2- 172.21.15.7:0/2172790457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f075413e000 0x7f075413d7c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7f075b037640 1 -- 172.21.15.7:0/2172790457 >> 172.21.15.7:0/2172790457 conn(0x7f07540b2ad0 msgr2=0x7f07540b2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:50.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7f075b037640 1 -- 172.21.15.7:0/2172790457 shutdown_connections 2024-03-28T16:15:50.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7f075b037640 1 -- 172.21.15.7:0/2172790457 wait complete. 2024-03-28T16:15:50.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7febc9e21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7febc9e21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7febc9e21640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-28T16:15:50.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7febc9620640 10 monclient(hunting): handle_auth_done global_id 4318 payload 293 2024-03-28T16:15:50.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7febc9620640 1 -- 172.21.15.7:0/2217813657 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7febb00047c0 msgr2=0x7febb0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:50.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7febc9620640 1 --2- 172.21.15.7:0/2217813657 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7febb00047c0 0x7febb0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7febc9620640 1 -- 172.21.15.7:0/2217813657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febb00052a0 msgr2=0x7febb0005950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:50.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7febc9620640 1 --2- 172.21.15.7:0/2217813657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febb00052a0 0x7febb0005950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7febc9620640 10 monclient: _finish_hunting 0 2024-03-28T16:15:50.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7febc9620640 1 monclient: found mon.b 2024-03-28T16:15:50.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7febc9620640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:50.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7febc9620640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:50.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7febc9e21640 1 -- 172.21.15.7:0/2217813657 reap_dead start 2024-03-28T16:15:50.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7febc9620640 1 -- 172.21.15.7:0/2217813657 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc40a8d90 con 0x7febb0003970 2024-03-28T16:15:50.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7febc9620640 10 monclient: _finish_auth 0 2024-03-28T16:15:50.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7febc9620640 10 monclient: _check_auth_tickets 2024-03-28T16:15:50.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7febc9620640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:50.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7febca622640 5 monclient: authenticate success, global_id 4318 2024-03-28T16:15:50.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7febca622640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:50.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7fd3122b0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd30c0a8b60 0x7fd30c12d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:58016/0 (socket says 172.21.15.7:58016) 2024-03-28T16:15:50.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7fd3122b0640 1 -- 172.21.15.7:0/4209453085 learned_addr learned my addr 172.21.15.7:0/4209453085 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:50.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7fd3122b0640 10 monclient(hunting): get_auth_request con 0x7fd30c0a8b60 auth_method 0 2024-03-28T16:15:50.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7fd3122b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7fd3122b0640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.668+0000 7fd3122b0640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.669+0000 7f51ccc06640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51c80b32e0 0x7f51c8143710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.669+0000 7f075b037640 10 monclient: build_initial_monmap 2024-03-28T16:15:50.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.669+0000 7f075b037640 10 monclient: monmap: 2024-03-28T16:15:50.740 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:50.740 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:50.740 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:50.740 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:50.740 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:50.740 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.740 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:50.740 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:50.740 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:50.740 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.669+0000 7f075b037640 1 librados: starting msgr at 2024-03-28T16:15:50.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.669+0000 7f075b037640 1 librados: starting objecter 2024-03-28T16:15:50.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.669+0000 7f075b037640 1 Processor -- start 2024-03-28T16:15:50.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.669+0000 7f075b037640 1 -- start start 2024-03-28T16:15:50.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.669+0000 7f075b037640 1 librados: setting wanted keys 2024-03-28T16:15:50.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.669+0000 7f075b037640 1 librados: calling monclient init 2024-03-28T16:15:50.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.669+0000 7f075b037640 10 monclient: init 2024-03-28T16:15:50.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f51ccc06640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51c80b32e0 0x7f51c8143710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:58032/0 (socket says 172.21.15.7:58032) 2024-03-28T16:15:50.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f51ccc06640 1 -- 172.21.15.7:0/753227592 learned_addr learned my addr 172.21.15.7:0/753227592 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:50.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f51ccc06640 10 monclient(hunting): get_auth_request con 0x7f51c80b32e0 auth_method 0 2024-03-28T16:15:50.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f51ccc06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f51ccc06640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f51ccc06640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f075b037640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:50.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f075b037640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-28T16:15:50.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f075b037640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f07540ada00 0x7f075413c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f075b037640 10 monclient(hunting): picked mon.noname-c con 0x7f07540ada00 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:50.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f075b037640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f075413d3e0 0x7f0754140380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f075b037640 10 monclient(hunting): picked mon.noname-a con 0x7f075413d3e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:50.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f075b037640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075413e000 0x7f07541353a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f075b037640 10 monclient(hunting): picked mon.noname-b con 0x7f075413e000 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:50.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f075b037640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f075b037640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075413d2a0 con 0x7f075413d3e0 2024-03-28T16:15:50.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f075b037640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f075b037640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075413fbf0 con 0x7f075413e000 2024-03-28T16:15:50.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f075b037640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f075b037640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07541408c0 con 0x7f07540ada00 2024-03-28T16:15:50.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f075b037640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:50.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.670+0000 7f075b037640 10 monclient(hunting): authenticate will time out at 1417.812728s 2024-03-28T16:15:50.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.671+0000 7f51ccc06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.671+0000 7f51ccc06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.671+0000 7f51ccc06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:50.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.671+0000 7f6c077fe640 1 -- 172.21.15.7:0/3234525566 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4031640 con 0x7f6c00032eb0 2024-03-28T16:15:50.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.671+0000 7f6c077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.671+0000 7f6c077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:50.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.671+0000 7f6c077fe640 10 monclient: dump: 2024-03-28T16:15:50.746 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.746 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.746 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.746 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.746 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.746 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.746 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.746 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.746 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.747 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.671+0000 7f6c077fe640 10 monclient: _finish_auth 0 2024-03-28T16:15:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.671+0000 7f6c077fe640 10 monclient: _check_auth_tickets 2024-03-28T16:15:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.671+0000 7f6c077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.671+0000 7f6c0ed37640 5 monclient: authenticate success, global_id 4303 2024-03-28T16:15:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.671+0000 7f6c0ed37640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.671+0000 7f6c077fe640 1 -- 172.21.15.7:0/3234525566 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bf40317a0 con 0x7f6c00032eb0 2024-03-28T16:15:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.671+0000 7f6c077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.671+0000 7f6c0ed37640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.671+0000 7f6c0ed37640 4 set_mon_vals no callback set 2024-03-28T16:15:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.672+0000 7f6c077fe640 1 -- 172.21.15.7:0/3234525566 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4031b30 con 0x7f6c00032eb0 2024-03-28T16:15:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.672+0000 7f6c077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.672+0000 7f6c077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.672+0000 7f6c077fe640 10 monclient: dump: 2024-03-28T16:15:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.748 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.672+0000 7f6c0ed37640 10 monclient: shutdown 2024-03-28T16:15:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.672+0000 7f6c0ed37640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.672+0000 7f6c0ed37640 1 -- 172.21.15.7:0/3234525566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c00032eb0 msgr2=0x7f6c000cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.672+0000 7f6c0ed37640 1 --2- 172.21.15.7:0/3234525566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c00032eb0 0x7f6c000cc060 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4009870 tx=0x7f6bf402b8f0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.672+0000 7f6c0ed37640 1 -- 172.21.15.7:0/3234525566 shutdown_connections 2024-03-28T16:15:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.672+0000 7f6c0ed37640 1 --2- 172.21.15.7:0/3234525566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c00032eb0 0x7f6c000cc060 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.672+0000 7f6c0ed37640 1 --2- 172.21.15.7:0/3234525566 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6c0002e1c0 0x7f6c0002c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.672+0000 7f6c0ed37640 1 -- 172.21.15.7:0/3234525566 >> 172.21.15.7:0/3234525566 conn(0x7f6c00036600 msgr2=0x7f6c00005ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.672+0000 7f6c0ed37640 1 -- 172.21.15.7:0/3234525566 shutdown_connections 2024-03-28T16:15:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.672+0000 7f6c0ed37640 1 -- 172.21.15.7:0/3234525566 wait complete. 2024-03-28T16:15:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f34697fa640 1 -- 172.21.15.7:0/3791127862 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346403e450 con 0x7f346c0ad840 2024-03-28T16:15:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f34697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f34697fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:50.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f34697fa640 10 monclient: dump: 2024-03-28T16:15:50.750 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.750 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.750 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.750 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.750 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.750 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.750 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.750 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.750 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.750 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f34697fa640 10 monclient: _finish_auth 0 2024-03-28T16:15:50.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f34697fa640 10 monclient: _check_auth_tickets 2024-03-28T16:15:50.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f34697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:50.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f34697fa640 1 -- 172.21.15.7:0/3791127862 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346403e5b0 con 0x7f346c0ad840 2024-03-28T16:15:50.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f34697fa640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:50.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f34697fa640 1 -- 172.21.15.7:0/3791127862 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346403e960 con 0x7f346c0ad840 2024-03-28T16:15:50.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f34697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f34697fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:15:50.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f34697fa640 10 monclient: dump: 2024-03-28T16:15:50.751 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.751 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.751 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.751 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.752 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.752 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.752 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.752 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.752 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.752 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f346affd640 4 set_mon_vals no callback set 2024-03-28T16:15:50.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f3473173640 5 monclient: authenticate success, global_id 4277 2024-03-28T16:15:50.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f3473173640 10 monclient: _renew_subs 2024-03-28T16:15:50.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f3473173640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:50.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f3473173640 1 -- 172.21.15.7:0/3791127862 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f346c0b2940 con 0x7f346c0ad840 2024-03-28T16:15:50.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f3473173640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:50.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f3473173640 10 monclient: _renew_subs 2024-03-28T16:15:50.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f3473173640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:50.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f3473173640 1 -- 172.21.15.7:0/3791127862 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f346c13de00 con 0x7f346c0ad840 2024-03-28T16:15:50.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f3473173640 1 librados: init done 2024-03-28T16:15:50.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f6c0ed37640 10 monclient: build_initial_monmap 2024-03-28T16:15:50.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f6c0ed37640 10 monclient: monmap: 2024-03-28T16:15:50.753 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:50.753 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:50.754 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:50.754 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:50.754 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:50.754 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.754 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:50.754 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:50.754 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:50.754 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f6c0ed37640 1 librados: starting msgr at 2024-03-28T16:15:50.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.673+0000 7f6c0ed37640 1 librados: starting objecter 2024-03-28T16:15:50.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f6c0ed37640 1 Processor -- start 2024-03-28T16:15:50.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f6c0ed37640 1 -- start start 2024-03-28T16:15:50.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f6c0ed37640 1 librados: setting wanted keys 2024-03-28T16:15:50.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f6c0ed37640 1 librados: calling monclient init 2024-03-28T16:15:50.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f6c0ed37640 10 monclient: init 2024-03-28T16:15:50.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f3473173640 10 client.4277.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:50.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f3473173640 15 client.4277.objecter _session_command_op_assign -1 1 2024-03-28T16:15:50.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f3473173640 20 client.4277.objecter _get_session osd=-1 returning homeless 2024-03-28T16:15:50.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f3473173640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:50.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f3473173640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:50.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f3473173640 1 -- 172.21.15.7:0/3791127862 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f346c0ade60 con 0x7f346c0ad840 2024-03-28T16:15:50.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f51ccc06640 10 monclient(hunting): handle_auth_done global_id 4321 payload 824 2024-03-28T16:15:50.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f51ccc06640 1 -- 172.21.15.7:0/753227592 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f51c813eab0 msgr2=0x7f51c80aba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f51ccc06640 1 --2- 172.21.15.7:0/753227592 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f51c813eab0 0x7f51c80aba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f51ccc06640 1 -- 172.21.15.7:0/753227592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51c80b3b90 msgr2=0x7f51c80acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f51ccc06640 1 --2- 172.21.15.7:0/753227592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51c80b3b90 0x7f51c80acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f51ccc06640 10 monclient: _finish_hunting 0 2024-03-28T16:15:50.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f51ccc06640 1 monclient: found mon.noname-b 2024-03-28T16:15:50.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f51ccc06640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:50.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f51ccc06640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:15:50.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f51ccc06640 1 -- 172.21.15.7:0/753227592 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c0009b30 con 0x7f51c80b32e0 2024-03-28T16:15:50.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7febc9620640 1 --2- 172.21.15.7:0/2217813657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febb0003970 0x7febb0003d70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7febc000d3c0 tx=0x7febb00057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5490b6967f44e6c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f6c0ed37640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:50.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f6c0ed37640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-28T16:15:50.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f6c0ed37640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6c0002e1c0 0x7f6c000cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f6c0ed37640 10 monclient(hunting): picked mon.noname-c con 0x7f6c0002e1c0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:50.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f6c0ed37640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c0002ec00 0x7f6c00029c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f6c0ed37640 10 monclient(hunting): picked mon.noname-b con 0x7f6c0002ec00 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:50.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f6c0ed37640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c00032eb0 0x7f6c000cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f6c0ed37640 10 monclient(hunting): picked mon.noname-a con 0x7f6c00032eb0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:50.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f6c0ed37640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f6c0ed37640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0002a190 con 0x7f6c00032eb0 2024-03-28T16:15:50.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f6c0ed37640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f6c0ed37640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c00029320 con 0x7f6c0002ec00 2024-03-28T16:15:50.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f6c0ed37640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f6c0ed37640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c000cf020 con 0x7f6c0002e1c0 2024-03-28T16:15:50.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f6c0ed37640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:50.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.674+0000 7f6c0ed37640 10 monclient(hunting): authenticate will time out at 1417.816815s 2024-03-28T16:15:50.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.677+0000 7f7a8affd640 1 -- 172.21.15.7:0/2692486868 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c031640 con 0x7f7a8c071920 2024-03-28T16:15:50.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.677+0000 7f7a8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.677+0000 7f7a8affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:15:50.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.677+0000 7f7a8affd640 10 monclient: dump: 2024-03-28T16:15:50.760 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.760 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.760 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.760 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.760 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.760 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.760 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.760 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.760 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.760 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.677+0000 7f7a8affd640 10 monclient: _finish_auth 0 2024-03-28T16:15:50.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.677+0000 7f7a8affd640 10 monclient: _check_auth_tickets 2024-03-28T16:15:50.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.677+0000 7f7a8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:50.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.677+0000 7f7a8affd640 1 -- 172.21.15.7:0/2692486868 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7c0317a0 con 0x7f7a8c071920 2024-03-28T16:15:50.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.677+0000 7f7a8affd640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:50.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.677+0000 7f7a8affd640 1 -- 172.21.15.7:0/2692486868 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c031b30 con 0x7f7a8c071920 2024-03-28T16:15:50.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.677+0000 7f7a9285c640 5 monclient: authenticate success, global_id 4311 2024-03-28T16:15:50.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.677+0000 7f7a9285c640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:50.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.677+0000 7f7a9285c640 4 set_mon_vals no callback set 2024-03-28T16:15:50.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.677+0000 7f7a8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.677+0000 7f7a8affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:15:50.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.677+0000 7f7a8affd640 10 monclient: dump: 2024-03-28T16:15:50.761 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.761 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.762 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.762 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.762 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.762 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.762 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.762 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.762 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.762 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.678+0000 7f7a9285c640 10 monclient: shutdown 2024-03-28T16:15:50.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.678+0000 7f7a9285c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:50.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.678+0000 7f7a9285c640 1 -- 172.21.15.7:0/2692486868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a8c071920 msgr2=0x7f7a8c13e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.678+0000 7f7a9285c640 1 --2- 172.21.15.7:0/2692486868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a8c071920 0x7f7a8c13e830 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c009870 tx=0x7f7a7c02b8f0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.678+0000 7f7a9285c640 1 -- 172.21.15.7:0/2692486868 shutdown_connections 2024-03-28T16:15:50.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.678+0000 7f7a9285c640 1 --2- 172.21.15.7:0/2692486868 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a8c0a89a0 0x7f7a8c13ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.678+0000 7f7a9285c640 1 --2- 172.21.15.7:0/2692486868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a8c071920 0x7f7a8c13e830 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.678+0000 7f7a9285c640 1 -- 172.21.15.7:0/2692486868 >> 172.21.15.7:0/2692486868 conn(0x7f7a8c0b3db0 msgr2=0x7f7a8c0b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:50.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.678+0000 7f51ccc06640 1 --2- 172.21.15.7:0/753227592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51c80b32e0 0x7f51c8143710 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f51bc00c820 tx=0x7f51bc00cce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.678+0000 7f51c57fa640 10 client.?.objecter ms_handle_connect 0x7f51c80b32e0 2024-03-28T16:15:50.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.678+0000 7f51c57fa640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:50.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.678+0000 7f7a9285c640 1 -- 172.21.15.7:0/2692486868 shutdown_connections 2024-03-28T16:15:50.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.678+0000 7f7a9285c640 1 -- 172.21.15.7:0/2692486868 wait complete. 2024-03-28T16:15:50.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.679+0000 7f7a9285c640 10 monclient: build_initial_monmap 2024-03-28T16:15:50.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.679+0000 7f7a9285c640 10 monclient: monmap: 2024-03-28T16:15:50.765 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:50.765 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:50.765 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:50.765 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:50.765 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:50.765 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.765 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:50.765 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:50.765 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:50.766 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.679+0000 7f7a9285c640 1 librados: starting msgr at 2024-03-28T16:15:50.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.679+0000 7f7a9285c640 1 librados: starting objecter 2024-03-28T16:15:50.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.679+0000 7f7a9285c640 1 Processor -- start 2024-03-28T16:15:50.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.679+0000 7f7a9285c640 1 -- start start 2024-03-28T16:15:50.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.679+0000 7f7a9285c640 1 librados: setting wanted keys 2024-03-28T16:15:50.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.679+0000 7f7a9285c640 1 librados: calling monclient init 2024-03-28T16:15:50.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.679+0000 7f7a9285c640 10 monclient: init 2024-03-28T16:15:50.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.680+0000 7f7a9285c640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:50.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.680+0000 7f7a9285c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-28T16:15:50.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.680+0000 7f7a9285c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a8c071920 0x7f7a8c141fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.680+0000 7f7a9285c640 10 monclient(hunting): picked mon.noname-c con 0x7f7a8c071920 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:50.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.680+0000 7f7a9285c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a8c0b1890 0x7f7a8c144ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.680+0000 7f7a9285c640 10 monclient(hunting): picked mon.noname-a con 0x7f7a8c0b1890 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:50.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.680+0000 7f7a9285c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a8c1450e0 0x7f7a8c1d9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.680+0000 7f7a9285c640 10 monclient(hunting): picked mon.noname-b con 0x7f7a8c1450e0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:50.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.680+0000 7f7a9285c640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.680+0000 7f7a9285c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c145590 con 0x7f7a8c0b1890 2024-03-28T16:15:50.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.680+0000 7f7a9285c640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.680+0000 7f7a9285c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c142850 con 0x7f7a8c1450e0 2024-03-28T16:15:50.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.680+0000 7f7a9285c640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.680+0000 7f7a9285c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c142b30 con 0x7f7a8c071920 2024-03-28T16:15:50.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.680+0000 7f7a9285c640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:50.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.680+0000 7f7a9285c640 10 monclient(hunting): authenticate will time out at 1417.822599s 2024-03-28T16:15:50.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.680+0000 7f5fa66c6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fa0149540 0x7f5fa0140700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.681+0000 7f5fa66c6640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fa0149540 0x7f5fa0140700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37880/0 (socket says 172.21.15.7:37880) 2024-03-28T16:15:50.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.681+0000 7f5fa66c6640 1 -- 172.21.15.7:0/3008119090 learned_addr learned my addr 172.21.15.7:0/3008119090 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:50.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.681+0000 7f5fa66c6640 10 monclient(hunting): get_auth_request con 0x7f5fa0149540 auth_method 0 2024-03-28T16:15:50.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.681+0000 7f5fa66c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.681+0000 7f5fa66c6640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.681+0000 7f5fa66c6640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.682+0000 7f0753fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f075413d3e0 0x7f0754140380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.682+0000 7f0753fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f075413d3e0 0x7f0754140380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37888/0 (socket says 172.21.15.7:37888) 2024-03-28T16:15:50.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.682+0000 7f0753fff640 1 -- 172.21.15.7:0/2546658315 learned_addr learned my addr 172.21.15.7:0/2546658315 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:50.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.682+0000 7f0753fff640 10 monclient(hunting): get_auth_request con 0x7f075413d3e0 auth_method 0 2024-03-28T16:15:50.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.682+0000 7f0753fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.682+0000 7f0753fff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.682+0000 7f0753fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.683+0000 7f5fa66c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.683+0000 7f5fa66c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.683+0000 7f5fa66c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:50.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.683+0000 7f0753fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.683+0000 7f0753fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.683+0000 7f0753fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:50.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.683+0000 7fd312ab1640 1 --2- 172.21.15.7:0/4209453085 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd30c0ab4b0 0x7fd30c126190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.684+0000 7febbbfff640 1 -- 172.21.15.7:0/2217813657 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc0033620 con 0x7febb0003970 2024-03-28T16:15:50.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.684+0000 7febbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.684+0000 7febbbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:50.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.684+0000 7febbbfff640 10 monclient: dump: 2024-03-28T16:15:50.771 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.771 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.771 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.771 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.772 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.772 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.772 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.772 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.772 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.772 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.684+0000 7febbbfff640 1 -- 172.21.15.7:0/2217813657 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc002f070 con 0x7febb0003970 2024-03-28T16:15:50.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.684+0000 7febca622640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:50.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.684+0000 7febbbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:50.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.684+0000 7febbbfff640 1 -- 172.21.15.7:0/2217813657 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc00339c0 con 0x7febb0003970 2024-03-28T16:15:50.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.684+0000 7febca622640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:50.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.684+0000 7febca622640 4 set_mon_vals no callback set 2024-03-28T16:15:50.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.684+0000 7febbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.684+0000 7febbbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:50.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.684+0000 7febbbfff640 10 monclient: dump: 2024-03-28T16:15:50.773 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.773 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.773 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.773 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.773 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.773 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.773 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.773 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.773 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.773 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.685+0000 7f07595d4640 1 --2- 172.21.15.7:0/2546658315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075413e000 0x7f07541353a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.685+0000 7febca622640 10 monclient: shutdown 2024-03-28T16:15:50.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.685+0000 7febca622640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:50.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.685+0000 7febca622640 1 -- 172.21.15.7:0/2217813657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febb0003970 msgr2=0x7febb0003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.685+0000 7febca622640 1 --2- 172.21.15.7:0/2217813657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febb0003970 0x7febb0003d70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7febc000d3c0 tx=0x7febb00057f0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.685+0000 7f5fa56c4640 1 --2- 172.21.15.7:0/3008119090 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5fa0071810 0x7f5fa0144a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.685+0000 7febca622640 1 -- 172.21.15.7:0/2217813657 shutdown_connections 2024-03-28T16:15:50.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.685+0000 7febca622640 1 --2- 172.21.15.7:0/2217813657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febb0003970 0x7febb0003d70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.685+0000 7febca622640 1 -- 172.21.15.7:0/2217813657 >> 172.21.15.7:0/2217813657 conn(0x7febc40b2cb0 msgr2=0x7febc40b3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:50.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.685+0000 7febca622640 1 -- 172.21.15.7:0/2217813657 shutdown_connections 2024-03-28T16:15:50.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.685+0000 7f07595d4640 10 monclient(hunting): get_auth_request con 0x7f075413e000 auth_method 0 2024-03-28T16:15:50.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.685+0000 7f07595d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.685+0000 7febca622640 1 -- 172.21.15.7:0/2217813657 wait complete. 2024-03-28T16:15:50.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.685+0000 7f07595d4640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.685+0000 7f07595d4640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.686+0000 7febca622640 10 monclient: build_initial_monmap 2024-03-28T16:15:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.686+0000 7febca622640 10 monclient: monmap: 2024-03-28T16:15:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:50.777 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.686+0000 7f5fa56c4640 10 monclient(hunting): get_auth_request con 0x7f5fa0071810 auth_method 0 2024-03-28T16:15:50.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.686+0000 7f5fa56c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.686+0000 7f5fa56c4640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.686+0000 7f5fa56c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.686+0000 7febca622640 1 librados: starting msgr at 2024-03-28T16:15:50.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.686+0000 7febca622640 1 librados: starting objecter 2024-03-28T16:15:50.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.686+0000 7f07595d4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.686+0000 7f07595d4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.686+0000 7febca622640 1 Processor -- start 2024-03-28T16:15:50.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.686+0000 7f07595d4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:50.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.686+0000 7febca622640 1 -- start start 2024-03-28T16:15:50.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.686+0000 7febca622640 1 librados: setting wanted keys 2024-03-28T16:15:50.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7febca622640 1 librados: calling monclient init 2024-03-28T16:15:50.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7febca622640 10 monclient: init 2024-03-28T16:15:50.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7f5fa56c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7f5fa56c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7f5fa56c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:50.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7fd3122b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7fd3122b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7fd3122b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:50.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7febca622640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:50.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7febca622640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-28T16:15:50.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7febca622640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7febb0003970 0x7febc40717a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7febca622640 10 monclient(hunting): picked mon.noname-c con 0x7febb0003970 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7febca622640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febc411a6b0 0x7febc411b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7febca622640 10 monclient(hunting): picked mon.noname-b con 0x7febc411a6b0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7febca622640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febc41d5a40 0x7febc41d5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7febca622640 10 monclient(hunting): picked mon.noname-a con 0x7febc41d5a40 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7febca622640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7f5fa66c6640 10 monclient(hunting): handle_auth_done global_id 4314 payload 824 2024-03-28T16:15:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7f5fa66c6640 1 -- 172.21.15.7:0/3008119090 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5fa006a490 msgr2=0x7f5fa0144400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7f5fa66c6640 1 --2- 172.21.15.7:0/3008119090 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5fa006a490 0x7f5fa0144400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7f5fa66c6640 1 -- 172.21.15.7:0/3008119090 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5fa0071810 msgr2=0x7f5fa0144a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7f5fa66c6640 1 --2- 172.21.15.7:0/3008119090 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5fa0071810 0x7f5fa0144a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7f5fa66c6640 10 monclient: _finish_hunting 0 2024-03-28T16:15:50.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7f5fa66c6640 1 monclient: found mon.noname-a 2024-03-28T16:15:50.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7f5fa66c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:50.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7f5fa66c6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:15:50.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7f5fa66c6640 1 -- 172.21.15.7:0/3008119090 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c009400 con 0x7f5fa0149540 2024-03-28T16:15:50.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7febca622640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc41d5520 con 0x7febc41d5a40 2024-03-28T16:15:50.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7febca622640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7febca622640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc4070960 con 0x7febc411a6b0 2024-03-28T16:15:50.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7febca622640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:50.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7febca622640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc406eb80 con 0x7febb0003970 2024-03-28T16:15:50.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.687+0000 7febca622640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:50.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.688+0000 7febca622640 10 monclient(hunting): authenticate will time out at 1417.829858s 2024-03-28T16:15:50.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.690+0000 7f07595d4640 10 monclient(hunting): handle_auth_done global_id 4324 payload 824 2024-03-28T16:15:50.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.690+0000 7f07595d4640 1 -- 172.21.15.7:0/2546658315 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f07540ada00 msgr2=0x7f075413c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.690+0000 7f07595d4640 1 --2- 172.21.15.7:0/2546658315 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f07540ada00 0x7f075413c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.690+0000 7f07595d4640 1 -- 172.21.15.7:0/2546658315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f075413d3e0 msgr2=0x7f0754140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.690+0000 7f07595d4640 1 --2- 172.21.15.7:0/2546658315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f075413d3e0 0x7f0754140380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.690+0000 7f07595d4640 10 monclient: _finish_hunting 0 2024-03-28T16:15:50.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.690+0000 7f07595d4640 1 monclient: found mon.noname-b 2024-03-28T16:15:50.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.690+0000 7f07595d4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:50.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.690+0000 7f07595d4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:15:50.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.690+0000 7f07595d4640 1 -- 172.21.15.7:0/2546658315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074400ae30 con 0x7f075413e000 2024-03-28T16:15:50.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.690+0000 7fd312ab1640 10 monclient(hunting): get_auth_request con 0x7fd30c0ab4b0 auth_method 0 2024-03-28T16:15:50.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.690+0000 7fd312ab1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.690+0000 7fd312ab1640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.690+0000 7fd312ab1640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.694+0000 7f5fa66c6640 1 --2- 172.21.15.7:0/3008119090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fa0149540 0x7f5fa0140700 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5f9800b990 tx=0x7f5f9800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.694+0000 7f5f967fc640 10 client.?.objecter ms_handle_connect 0x7f5fa0149540 2024-03-28T16:15:50.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.694+0000 7f5f967fc640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:50.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.695+0000 7f7a8b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a8c0b1890 0x7f7a8c144ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.696+0000 7fd3122b0640 10 monclient(hunting): handle_auth_done global_id 4330 payload 824 2024-03-28T16:15:50.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.696+0000 7fd3122b0640 1 -- 172.21.15.7:0/4209453085 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd30c0a9dc0 msgr2=0x7fd30c12dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.696+0000 7fd3122b0640 1 --2- 172.21.15.7:0/4209453085 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd30c0a9dc0 0x7fd30c12dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.696+0000 7fd3122b0640 1 -- 172.21.15.7:0/4209453085 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd30c0ab4b0 msgr2=0x7fd30c126190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.696+0000 7fd3122b0640 1 --2- 172.21.15.7:0/4209453085 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd30c0ab4b0 0x7fd30c126190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.696+0000 7fd3122b0640 10 monclient: _finish_hunting 0 2024-03-28T16:15:50.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.696+0000 7fd3122b0640 1 monclient: found mon.noname-b 2024-03-28T16:15:50.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.696+0000 7fd3122b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:50.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.696+0000 7fd3122b0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:15:50.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.696+0000 7fd3122b0640 1 -- 172.21.15.7:0/4209453085 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd304009b30 con 0x7fd30c0a8b60 2024-03-28T16:15:50.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51c57fa640 1 -- 172.21.15.7:0/753227592 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51bc004220 con 0x7f51c80b32e0 2024-03-28T16:15:50.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51c57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51c57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:50.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51c57fa640 10 monclient: dump: 2024-03-28T16:15:50.792 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.792 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.792 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.792 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.792 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.792 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.792 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.792 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.793 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.793 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51c57fa640 10 monclient: _finish_auth 0 2024-03-28T16:15:50.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51c57fa640 10 monclient: _check_auth_tickets 2024-03-28T16:15:50.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51c57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:50.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51c57fa640 1 -- 172.21.15.7:0/753227592 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51bc004380 con 0x7f51c80b32e0 2024-03-28T16:15:50.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51c57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:50.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51c6ffd640 4 set_mon_vals no callback set 2024-03-28T16:15:50.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51c57fa640 1 -- 172.21.15.7:0/753227592 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51bc0079f0 con 0x7f51c80b32e0 2024-03-28T16:15:50.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51c57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51c57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:50.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51c57fa640 10 monclient: dump: 2024-03-28T16:15:50.793 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.794 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.794 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.794 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.794 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.794 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.794 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.794 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.794 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.794 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51cdc08640 5 monclient: authenticate success, global_id 4321 2024-03-28T16:15:50.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51cdc08640 10 monclient: _renew_subs 2024-03-28T16:15:50.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51cdc08640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:50.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51cdc08640 1 -- 172.21.15.7:0/753227592 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c80ac240 con 0x7f51c80b32e0 2024-03-28T16:15:50.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51cdc08640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:50.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51cdc08640 10 monclient: _renew_subs 2024-03-28T16:15:50.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51cdc08640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:50.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51cdc08640 1 -- 172.21.15.7:0/753227592 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c81d6560 con 0x7f51c80b32e0 2024-03-28T16:15:50.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.698+0000 7f51cdc08640 1 librados: init done 2024-03-28T16:15:50.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.699+0000 7f6c07fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c0002ec00 0x7f6c00029c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.699+0000 7f51cdc08640 10 client.4321.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:50.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.699+0000 7f51cdc08640 15 client.4321.objecter _session_command_op_assign -1 1 2024-03-28T16:15:50.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.699+0000 7f51cdc08640 20 client.4321.objecter _get_session osd=-1 returning homeless 2024-03-28T16:15:50.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.699+0000 7f51cdc08640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:50.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.699+0000 7f51cdc08640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:50.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.699+0000 7f51cdc08640 1 -- 172.21.15.7:0/753227592 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f51c80670c0 con 0x7f51c80b32e0 2024-03-28T16:15:50.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.700+0000 7f6c07fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c0002ec00 0x7f6c00029c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:58062/0 (socket says 172.21.15.7:58062) 2024-03-28T16:15:50.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.700+0000 7f6c07fff640 1 -- 172.21.15.7:0/312124861 learned_addr learned my addr 172.21.15.7:0/312124861 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:50.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.700+0000 7f6c07fff640 10 monclient(hunting): get_auth_request con 0x7f6c0002ec00 auth_method 0 2024-03-28T16:15:50.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.700+0000 7f6c07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.700+0000 7f6c07fff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.700+0000 7f6c07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.700+0000 7f7a8b7fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a8c0b1890 0x7f7a8c144ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37902/0 (socket says 172.21.15.7:37902) 2024-03-28T16:15:50.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.700+0000 7f7a8b7fe640 1 -- 172.21.15.7:0/1766437136 learned_addr learned my addr 172.21.15.7:0/1766437136 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:50.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.700+0000 7f7a8b7fe640 10 monclient(hunting): get_auth_request con 0x7f7a8c0b1890 auth_method 0 2024-03-28T16:15:50.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.700+0000 7f7a8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.700+0000 7f7a8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.700+0000 7f7a8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.701+0000 7f6c0d2d4640 1 --2- 172.21.15.7:0/312124861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c00032eb0 0x7f6c000cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.701+0000 7f34697fa640 1 -- 172.21.15.7:0/3791127862 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f346403eac0 con 0x7f346c0ad840 2024-03-28T16:15:50.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.701+0000 7f07595d4640 1 --2- 172.21.15.7:0/2546658315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075413e000 0x7f07541353a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f074c00c7b0 tx=0x7f074c00cc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.701+0000 7f07517fa640 10 client.?.objecter ms_handle_connect 0x7f075413e000 2024-03-28T16:15:50.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.701+0000 7f07517fa640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:50.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.701+0000 7f34697fa640 1 --2- 172.21.15.7:0/3791127862 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f345806c590 0x7f345806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.702+0000 7f3470f0f640 1 --2- 172.21.15.7:0/3791127862 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f345806c590 0x7f345806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.702+0000 7f3470f0f640 10 monclient: get_auth_request con 0x7f345806c590 auth_method 0 2024-03-28T16:15:50.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.702+0000 7fd3122b0640 1 --2- 172.21.15.7:0/4209453085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd30c0a8b60 0x7fd30c12d7d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd3040093e0 tx=0x7fd304004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.702+0000 7fd302ffd640 10 client.?.objecter ms_handle_connect 0x7fd30c0a8b60 2024-03-28T16:15:50.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.702+0000 7fd302ffd640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:50.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.702+0000 7febc9e21640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febc41d5a40 0x7febc41d5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.702+0000 7f7a8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.702+0000 7f7a8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.702+0000 7f7a8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:50.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.702+0000 7f3470f0f640 1 --2- 172.21.15.7:0/3791127862 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f345806c590 0x7f345806ea50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f345c009710 tx=0x7f345c006c90 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.702+0000 7f34697fa640 10 client.4277.objecter ms_handle_connect 0x7f345806c590 2024-03-28T16:15:50.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.703+0000 7f6c07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.703+0000 7f6c07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.703+0000 7f6c07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:50.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.705+0000 7f34697fa640 1 -- 172.21.15.7:0/3791127862 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f346407bf80 con 0x7f346c0ad840 2024-03-28T16:15:50.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.705+0000 7f34697fa640 10 client.4277.objecter ms_dispatch 0x7f346c000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.705+0000 7f34697fa640 3 client.4277.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.705+0000 7f34697fa640 3 client.4277.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.705+0000 7f34697fa640 10 client.4277.objecter checking command 1 2024-03-28T16:15:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.705+0000 7f34697fa640 1 --2- 172.21.15.7:0/3791127862 >> [v2:172.21.15.7:6816/34921,v1:172.21.15.7:6817/34921] conn(0x7f3458072970 0x7f3458074d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.705+0000 7f34697fa640 20 client.4277.objecter _get_session s=0x7f34580722a0 osd=3 3 2024-03-28T16:15:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.705+0000 7f34697fa640 20 client.4277.objecter put_session s=0x7f34580722a0 osd=3 3 2024-03-28T16:15:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.705+0000 7f34697fa640 15 client.4277.objecter _session_command_op_remove -1 1 2024-03-28T16:15:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.705+0000 7f34697fa640 20 client.4277.objecter _get_session s=0x7f34580722a0 osd=3 3 2024-03-28T16:15:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.705+0000 7f34697fa640 20 client.4277.objecter get_session s=0x7f34580722a0 osd=3 3 2024-03-28T16:15:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.705+0000 7f34697fa640 15 client.4277.objecter _session_command_op_assign 3 1 2024-03-28T16:15:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.705+0000 7f34697fa640 20 client.4277.objecter put_session s=0x7f34580722a0 osd=3 4 2024-03-28T16:15:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.705+0000 7f34697fa640 10 client.4277.objecter _send_command 1 2024-03-28T16:15:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.705+0000 7f34697fa640 1 -- 172.21.15.7:0/3791127862 --> [v2:172.21.15.7:6816/34921,v1:172.21.15.7:6817/34921] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3458075440 con 0x7f3458072970 2024-03-28T16:15:50.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.705+0000 7f34697fa640 20 client.4277.objecter dump_active .. 0 homeless 2024-03-28T16:15:50.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.705+0000 7f34697fa640 1 -- 172.21.15.7:0/3791127862 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3464077bc0 con 0x7f346c0ad840 2024-03-28T16:15:50.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.705+0000 7f34697fa640 10 monclient: handle_get_version_reply finishing 1 version 28 2024-03-28T16:15:50.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f3471710640 1 --2- 172.21.15.7:0/3791127862 >> [v2:172.21.15.7:6816/34921,v1:172.21.15.7:6817/34921] conn(0x7f3458072970 0x7f3458074d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f7a8b7fe640 10 monclient(hunting): handle_auth_done global_id 4323 payload 824 2024-03-28T16:15:50.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f7a8b7fe640 1 -- 172.21.15.7:0/1766437136 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a8c071920 msgr2=0x7f7a8c141fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f7a8b7fe640 1 --2- 172.21.15.7:0/1766437136 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a8c071920 0x7f7a8c141fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f7a8b7fe640 1 -- 172.21.15.7:0/1766437136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a8c1450e0 msgr2=0x7f7a8c1d9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f7a8b7fe640 1 --2- 172.21.15.7:0/1766437136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7a8c1450e0 0x7f7a8c1d9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f7a8b7fe640 10 monclient: _finish_hunting 0 2024-03-28T16:15:50.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f7a8b7fe640 1 monclient: found mon.noname-a 2024-03-28T16:15:50.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f7a8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:50.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f7a8b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:15:50.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f7a8b7fe640 1 -- 172.21.15.7:0/1766437136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7c0094d0 con 0x7f7a8c0b1890 2024-03-28T16:15:50.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f6c07fff640 10 monclient(hunting): handle_auth_done global_id 4333 payload 824 2024-03-28T16:15:50.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f6c07fff640 1 -- 172.21.15.7:0/312124861 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6c0002e1c0 msgr2=0x7f6c000cacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f6c07fff640 1 --2- 172.21.15.7:0/312124861 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6c0002e1c0 0x7f6c000cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f6c07fff640 1 -- 172.21.15.7:0/312124861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c00032eb0 msgr2=0x7f6c000cf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f6c07fff640 1 --2- 172.21.15.7:0/312124861 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6c00032eb0 0x7f6c000cf4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f6c07fff640 10 monclient: _finish_hunting 0 2024-03-28T16:15:50.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f6c07fff640 1 monclient: found mon.noname-b 2024-03-28T16:15:50.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f6c07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:50.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f6c07fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:15:50.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f6c07fff640 1 -- 172.21.15.7:0/312124861 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf40094d0 con 0x7f6c0002ec00 2024-03-28T16:15:50.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7febc9e21640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febc41d5a40 0x7febc41d5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37918/0 (socket says 172.21.15.7:37918) 2024-03-28T16:15:50.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7febc9e21640 1 -- 172.21.15.7:0/3386136446 learned_addr learned my addr 172.21.15.7:0/3386136446 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:50.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7febc9e21640 10 monclient(hunting): get_auth_request con 0x7febc41d5a40 auth_method 0 2024-03-28T16:15:50.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7febc9e21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7febc9e21640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7febc9e21640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.706+0000 7f3471710640 10 monclient: get_auth_request con 0x7f3458072970 auth_method 0 2024-03-28T16:15:50.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.708+0000 7febc9e21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:50.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.708+0000 7febc9e21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:50.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.708+0000 7febc9e21640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:50.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.710+0000 7f3471710640 1 --2- 172.21.15.7:0/3791127862 >> [v2:172.21.15.7:6816/34921,v1:172.21.15.7:6817/34921] conn(0x7f3458072970 0x7f3458074d90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.710+0000 7f34697fa640 10 client.4277.objecter ms_handle_connect 0x7f3458072970 2024-03-28T16:15:50.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.712+0000 7febc9620640 1 --2- 172.21.15.7:0/3386136446 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7febb0003970 0x7febc40717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.712+0000 7febc9620640 10 monclient(hunting): get_auth_request con 0x7febb0003970 auth_method 0 2024-03-28T16:15:50.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.712+0000 7febc9620640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.712+0000 7febc9620640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.712+0000 7febc9620640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.713+0000 7febc9e21640 10 monclient(hunting): handle_auth_done global_id 4326 payload 824 2024-03-28T16:15:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.713+0000 7febc9e21640 1 -- 172.21.15.7:0/3386136446 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7febb0003970 msgr2=0x7febc40717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.713+0000 7febc9e21640 1 --2- 172.21.15.7:0/3386136446 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7febb0003970 0x7febc40717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.713+0000 7febc9e21640 1 -- 172.21.15.7:0/3386136446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febc411a6b0 msgr2=0x7febc411b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.713+0000 7febc9e21640 1 --2- 172.21.15.7:0/3386136446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febc411a6b0 0x7febc411b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.713+0000 7febc9e21640 10 monclient: _finish_hunting 0 2024-03-28T16:15:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.713+0000 7febc9e21640 1 monclient: found mon.noname-a 2024-03-28T16:15:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.713+0000 7febc9e21640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.713+0000 7febc9e21640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:15:50.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.713+0000 7febc9e21640 1 -- 172.21.15.7:0/3386136446 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc406ed40 con 0x7febc41d5a40 2024-03-28T16:15:50.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5f967fc640 1 -- 172.21.15.7:0/3008119090 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9800cba0 con 0x7f5fa0149540 2024-03-28T16:15:50.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5f967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5f967fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:15:50.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5f967fc640 10 monclient: dump: 2024-03-28T16:15:50.811 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.811 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.812 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.812 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.812 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.812 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.812 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.812 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.812 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.812 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5f967fc640 10 monclient: _finish_auth 0 2024-03-28T16:15:50.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5f967fc640 10 monclient: _check_auth_tickets 2024-03-28T16:15:50.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5f967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:50.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5f967fc640 1 -- 172.21.15.7:0/3008119090 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9800cd00 con 0x7f5fa0149540 2024-03-28T16:15:50.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5f967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:50.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5f967fc640 1 -- 172.21.15.7:0/3008119090 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98014680 con 0x7f5fa0149540 2024-03-28T16:15:50.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5f967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5f967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:15:50.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5f967fc640 10 monclient: dump: 2024-03-28T16:15:50.813 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.813 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.813 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.813 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.813 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.813 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.813 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.813 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.813 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.813 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5f97fff640 4 set_mon_vals no callback set 2024-03-28T16:15:50.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f7a88ff9640 10 client.?.objecter ms_handle_connect 0x7f7a8c0b1890 2024-03-28T16:15:50.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f7a88ff9640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:50.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f7a8b7fe640 1 --2- 172.21.15.7:0/1766437136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a8c0b1890 0x7f7a8c144ba0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000b500 tx=0x7f7a8000b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5fa6ec7640 5 monclient: authenticate success, global_id 4314 2024-03-28T16:15:50.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5fa6ec7640 10 monclient: _renew_subs 2024-03-28T16:15:50.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5fa6ec7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:50.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5fa6ec7640 1 -- 172.21.15.7:0/3008119090 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa0145680 con 0x7f5fa0149540 2024-03-28T16:15:50.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5fa6ec7640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:50.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5fa6ec7640 10 monclient: _renew_subs 2024-03-28T16:15:50.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5fa6ec7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:50.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.717+0000 7f5fa6ec7640 1 -- 172.21.15.7:0/3008119090 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa0140e20 con 0x7f5fa0149540 2024-03-28T16:15:50.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.718+0000 7f5fa6ec7640 1 librados: init done 2024-03-28T16:15:50.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.718+0000 7febb9ffb640 10 client.?.objecter ms_handle_connect 0x7febc41d5a40 2024-03-28T16:15:50.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.718+0000 7febb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:50.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.718+0000 7febc9e21640 1 --2- 172.21.15.7:0/3386136446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febc41d5a40 0x7febc41d5ef0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7febc40ad070 tx=0x7febbc00fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.718+0000 7f5fa6ec7640 10 client.4314.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:50.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.718+0000 7f5fa6ec7640 15 client.4314.objecter _session_command_op_assign -1 1 2024-03-28T16:15:50.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.718+0000 7f5fa6ec7640 20 client.4314.objecter _get_session osd=-1 returning homeless 2024-03-28T16:15:50.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.718+0000 7f5fa6ec7640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:50.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.718+0000 7f5fa6ec7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:50.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.718+0000 7f5fa6ec7640 1 -- 172.21.15.7:0/3008119090 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5fa0146680 con 0x7f5fa0149540 2024-03-28T16:15:50.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.719+0000 7fd302ffd640 1 -- 172.21.15.7:0/4209453085 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30403e450 con 0x7fd30c0a8b60 2024-03-28T16:15:50.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd302ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd302ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:50.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd302ffd640 10 monclient: dump: 2024-03-28T16:15:50.819 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.819 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.820 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.820 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.820 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.820 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.820 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.820 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.820 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.820 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd302ffd640 10 monclient: _finish_auth 0 2024-03-28T16:15:50.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd302ffd640 10 monclient: _check_auth_tickets 2024-03-28T16:15:50.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd302ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:50.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd314514640 5 monclient: authenticate success, global_id 4330 2024-03-28T16:15:50.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd314514640 10 monclient: _renew_subs 2024-03-28T16:15:50.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd314514640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:50.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd314514640 1 -- 172.21.15.7:0/4209453085 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd30c1266d0 con 0x7fd30c0a8b60 2024-03-28T16:15:50.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd314514640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:50.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd314514640 10 monclient: _renew_subs 2024-03-28T16:15:50.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd314514640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:50.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd314514640 1 -- 172.21.15.7:0/4209453085 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd30c126bb0 con 0x7fd30c0a8b60 2024-03-28T16:15:50.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd314514640 1 librados: init done 2024-03-28T16:15:50.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd302ffd640 1 -- 172.21.15.7:0/4209453085 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30403e5d0 con 0x7fd30c0a8b60 2024-03-28T16:15:50.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd302ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:50.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd310aad640 4 set_mon_vals no callback set 2024-03-28T16:15:50.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd314514640 10 client.4330.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:50.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd302ffd640 1 -- 172.21.15.7:0/4209453085 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30403e980 con 0x7fd30c0a8b60 2024-03-28T16:15:50.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd302ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd302ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:50.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd302ffd640 10 monclient: dump: 2024-03-28T16:15:50.822 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.822 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.822 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.822 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.822 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.822 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.822 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.822 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.822 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.822 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd314514640 15 client.4330.objecter _session_command_op_assign -1 1 2024-03-28T16:15:50.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd314514640 20 client.4330.objecter _get_session osd=-1 returning homeless 2024-03-28T16:15:50.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.720+0000 7fd314514640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:50.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.721+0000 7fd314514640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:50.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.721+0000 7fd314514640 1 -- 172.21.15.7:0/4209453085 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd2f40028a0 con 0x7fd30c0a8b60 2024-03-28T16:15:50.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.731+0000 7f51c57fa640 1 -- 172.21.15.7:0/753227592 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f51bc007b50 con 0x7f51c80b32e0 2024-03-28T16:15:50.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.732+0000 7f51c57fa640 1 --2- 172.21.15.7:0/753227592 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f51ac06c5c0 0x7f51ac06ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.732+0000 7f51c7fff640 1 --2- 172.21.15.7:0/753227592 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f51ac06c5c0 0x7f51ac06ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.732+0000 7f51c7fff640 10 monclient: get_auth_request con 0x7f51ac06c5c0 auth_method 0 2024-03-28T16:15:50.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.732+0000 7f51c57fa640 10 client.4321.objecter ms_handle_connect 0x7f51ac06c5c0 2024-03-28T16:15:50.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.732+0000 7f51c7fff640 1 --2- 172.21.15.7:0/753227592 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f51ac06c5c0 0x7f51ac06ea80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f51c81443b0 tx=0x7f51b4006c90 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.733+0000 7f51c57fa640 1 -- 172.21.15.7:0/753227592 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f51bc058c10 con 0x7f51c80b32e0 2024-03-28T16:15:50.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.733+0000 7f51c57fa640 10 client.4321.objecter ms_dispatch 0x7f51c8000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:50.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.733+0000 7f51c57fa640 3 client.4321.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:50.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.733+0000 7f51c57fa640 3 client.4321.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:50.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.733+0000 7f51c57fa640 10 client.4321.objecter checking command 1 2024-03-28T16:15:50.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.733+0000 7f51c57fa640 1 --2- 172.21.15.7:0/753227592 >> [v2:172.21.15.83:6802/35295,v1:172.21.15.83:6803/35295] conn(0x7f51ac0729a0 0x7f51ac074dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.733+0000 7f51c57fa640 20 client.4321.objecter _get_session s=0x7f51ac0722d0 osd=7 3 2024-03-28T16:15:50.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.733+0000 7f51c57fa640 20 client.4321.objecter put_session s=0x7f51ac0722d0 osd=7 3 2024-03-28T16:15:50.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.733+0000 7f51c57fa640 15 client.4321.objecter _session_command_op_remove -1 1 2024-03-28T16:15:50.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.734+0000 7f51c57fa640 20 client.4321.objecter _get_session s=0x7f51ac0722d0 osd=7 3 2024-03-28T16:15:50.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.734+0000 7f51c57fa640 20 client.4321.objecter get_session s=0x7f51ac0722d0 osd=7 3 2024-03-28T16:15:50.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.734+0000 7f51c57fa640 15 client.4321.objecter _session_command_op_assign 7 1 2024-03-28T16:15:50.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.734+0000 7f51c57fa640 20 client.4321.objecter put_session s=0x7f51ac0722d0 osd=7 4 2024-03-28T16:15:50.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.734+0000 7f51c57fa640 10 client.4321.objecter _send_command 1 2024-03-28T16:15:50.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.734+0000 7f51c57fa640 1 -- 172.21.15.7:0/753227592 --> [v2:172.21.15.83:6802/35295,v1:172.21.15.83:6803/35295] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f51ac075470 con 0x7f51ac0729a0 2024-03-28T16:15:50.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.734+0000 7f6c07fff640 1 --2- 172.21.15.7:0/312124861 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c0002ec00 0x7f6c00029c50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc00c410 tx=0x7f6bfc00c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.734+0000 7f51c57fa640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-28T16:15:50.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.734+0000 7f6c057fa640 10 client.?.objecter ms_handle_connect 0x7f6c0002ec00 2024-03-28T16:15:50.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.734+0000 7f6c057fa640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:50.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.734+0000 7f51cd407640 1 --2- 172.21.15.7:0/753227592 >> [v2:172.21.15.83:6802/35295,v1:172.21.15.83:6803/35295] conn(0x7f51ac0729a0 0x7f51ac074dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.734+0000 7f51c57fa640 1 -- 172.21.15.7:0/753227592 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51bc054850 con 0x7f51c80b32e0 2024-03-28T16:15:50.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.734+0000 7f51c57fa640 10 monclient: handle_get_version_reply finishing 1 version 28 2024-03-28T16:15:50.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.734+0000 7f51cd407640 10 monclient: get_auth_request con 0x7f51ac0729a0 auth_method 0 2024-03-28T16:15:50.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.738+0000 7f51cd407640 1 --2- 172.21.15.7:0/753227592 >> [v2:172.21.15.83:6802/35295,v1:172.21.15.83:6803/35295] conn(0x7f51ac0729a0 0x7f51ac074dc0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.738+0000 7f51c57fa640 10 client.4321.objecter ms_handle_connect 0x7f51ac0729a0 2024-03-28T16:15:50.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.742+0000 7f5f967fc640 1 -- 172.21.15.7:0/3008119090 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5f9801d3b0 con 0x7f5fa0149540 2024-03-28T16:15:50.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.742+0000 7f5f967fc640 1 --2- 172.21.15.7:0/3008119090 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f5f8406c5e0 0x7f5f8406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.743+0000 7f5fa5ec5640 1 --2- 172.21.15.7:0/3008119090 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f5f8406c5e0 0x7f5f8406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.743+0000 7f5fa5ec5640 10 monclient: get_auth_request con 0x7f5f8406c5e0 auth_method 0 2024-03-28T16:15:50.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.743+0000 7f5fa5ec5640 1 --2- 172.21.15.7:0/3008119090 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f5f8406c5e0 0x7f5f8406eaa0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c034000 tx=0x7f5f9c02f000 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.743+0000 7f5f967fc640 10 client.4314.objecter ms_handle_connect 0x7f5f8406c5e0 2024-03-28T16:15:50.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.749+0000 7f5f967fc640 1 -- 172.21.15.7:0/3008119090 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5f98058980 con 0x7f5fa0149540 2024-03-28T16:15:50.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.749+0000 7f5f967fc640 10 client.4314.objecter ms_dispatch 0x7f5fa0000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:50.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.749+0000 7f5f967fc640 3 client.4314.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:50.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.749+0000 7f5f967fc640 3 client.4314.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:50.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.749+0000 7f5f967fc640 10 client.4314.objecter checking command 1 2024-03-28T16:15:50.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.749+0000 7f5f967fc640 1 --2- 172.21.15.7:0/3008119090 >> [v2:172.21.15.83:6826/35296,v1:172.21.15.83:6827/35296] conn(0x7f5f840729c0 0x7f5f84074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.749+0000 7f5f967fc640 20 client.4314.objecter _get_session s=0x7f5f840722f0 osd=5 3 2024-03-28T16:15:50.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.749+0000 7f5f967fc640 20 client.4314.objecter put_session s=0x7f5f840722f0 osd=5 3 2024-03-28T16:15:50.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.749+0000 7f5f967fc640 15 client.4314.objecter _session_command_op_remove -1 1 2024-03-28T16:15:50.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.749+0000 7f5f967fc640 20 client.4314.objecter _get_session s=0x7f5f840722f0 osd=5 3 2024-03-28T16:15:50.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.749+0000 7f5f967fc640 20 client.4314.objecter get_session s=0x7f5f840722f0 osd=5 3 2024-03-28T16:15:50.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.749+0000 7f5f967fc640 15 client.4314.objecter _session_command_op_assign 5 1 2024-03-28T16:15:50.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.749+0000 7f5f967fc640 20 client.4314.objecter put_session s=0x7f5f840722f0 osd=5 4 2024-03-28T16:15:50.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.749+0000 7f5f967fc640 10 client.4314.objecter _send_command 1 2024-03-28T16:15:50.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.749+0000 7f5f967fc640 1 -- 172.21.15.7:0/3008119090 --> [v2:172.21.15.83:6826/35296,v1:172.21.15.83:6827/35296] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f84075490 con 0x7f5f840729c0 2024-03-28T16:15:50.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.749+0000 7f5f967fc640 20 client.4314.objecter dump_active .. 0 homeless 2024-03-28T16:15:50.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.749+0000 7f5f967fc640 1 -- 172.21.15.7:0/3008119090 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f98054cc0 con 0x7f5fa0149540 2024-03-28T16:15:50.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.749+0000 7f5f967fc640 10 monclient: handle_get_version_reply finishing 1 version 28 2024-03-28T16:15:50.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.750+0000 7f5fa56c4640 1 --2- 172.21.15.7:0/3008119090 >> [v2:172.21.15.83:6826/35296,v1:172.21.15.83:6827/35296] conn(0x7f5f840729c0 0x7f5f84074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.750+0000 7f5fa56c4640 10 monclient: get_auth_request con 0x7f5f840729c0 auth_method 0 2024-03-28T16:15:50.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febb9ffb640 1 -- 172.21.15.7:0/3386136446 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febbc006b30 con 0x7febc41d5a40 2024-03-28T16:15:50.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febb9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:15:50.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febb9ffb640 10 monclient: dump: 2024-03-28T16:15:50.833 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.833 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.833 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.834 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.834 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.834 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.834 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.834 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.834 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.834 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febb9ffb640 10 monclient: _finish_auth 0 2024-03-28T16:15:50.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febb9ffb640 10 monclient: _check_auth_tickets 2024-03-28T16:15:50.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:50.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febca622640 5 monclient: authenticate success, global_id 4326 2024-03-28T16:15:50.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febca622640 10 monclient: _renew_subs 2024-03-28T16:15:50.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febca622640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:50.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febca622640 1 -- 172.21.15.7:0/3386136446 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc4138d30 con 0x7febc41d5a40 2024-03-28T16:15:50.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febb9ffb640 1 -- 172.21.15.7:0/3386136446 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febbc006c90 con 0x7febc41d5a40 2024-03-28T16:15:50.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febb9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:50.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febb9ffb640 1 -- 172.21.15.7:0/3386136446 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febbc006b30 con 0x7febc41d5a40 2024-03-28T16:15:50.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febb9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:15:50.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febb9ffb640 10 monclient: dump: 2024-03-28T16:15:50.835 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.835 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.835 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.835 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.835 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.836 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.836 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.836 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.836 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.836 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febbb7fe640 4 set_mon_vals no callback set 2024-03-28T16:15:50.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febca622640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:50.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febca622640 10 monclient: _renew_subs 2024-03-28T16:15:50.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febca622640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:50.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febca622640 1 -- 172.21.15.7:0/3386136446 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febc41391f0 con 0x7febc41d5a40 2024-03-28T16:15:50.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.751+0000 7febca622640 1 librados: init done 2024-03-28T16:15:50.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.752+0000 7febca622640 10 client.4326.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:50.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.752+0000 7febca622640 15 client.4326.objecter _session_command_op_assign -1 1 2024-03-28T16:15:50.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.752+0000 7febca622640 20 client.4326.objecter _get_session osd=-1 returning homeless 2024-03-28T16:15:50.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.752+0000 7febca622640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:50.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.752+0000 7febca622640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:50.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.752+0000 7febca622640 1 -- 172.21.15.7:0/3386136446 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7febc406bab0 con 0x7febc41d5a40 2024-03-28T16:15:50.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.752+0000 7f7a88ff9640 1 -- 172.21.15.7:0/1766437136 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80010870 con 0x7f7a8c0b1890 2024-03-28T16:15:50.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.752+0000 7f7a88ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.752+0000 7f7a88ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:15:50.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.753+0000 7f7a88ff9640 10 monclient: dump: 2024-03-28T16:15:50.838 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.838 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.838 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.838 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.838 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.838 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.839 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.839 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.839 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.839 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.753+0000 7f7a88ff9640 10 monclient: _finish_auth 0 2024-03-28T16:15:50.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.753+0000 7f7a88ff9640 10 monclient: _check_auth_tickets 2024-03-28T16:15:50.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.753+0000 7f7a88ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:50.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.753+0000 7f7a9285c640 5 monclient: authenticate success, global_id 4323 2024-03-28T16:15:50.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.753+0000 7f7a9285c640 10 monclient: _renew_subs 2024-03-28T16:15:50.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.753+0000 7f7a9285c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:50.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.753+0000 7f7a9285c640 1 -- 172.21.15.7:0/1766437136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8c1440e0 con 0x7f7a8c0b1890 2024-03-28T16:15:50.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.753+0000 7f7a88ff9640 1 -- 172.21.15.7:0/1766437136 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a800109d0 con 0x7f7a8c0b1890 2024-03-28T16:15:50.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.753+0000 7f7a9285c640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:50.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.753+0000 7f7a88ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:50.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.753+0000 7f7a8a7fc640 4 set_mon_vals no callback set 2024-03-28T16:15:50.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.753+0000 7f7a9285c640 10 monclient: _renew_subs 2024-03-28T16:15:50.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.753+0000 7f7a9285c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:50.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.753+0000 7f7a9285c640 1 -- 172.21.15.7:0/1766437136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a8c141170 con 0x7f7a8c0b1890 2024-03-28T16:15:50.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.753+0000 7f7a9285c640 1 librados: init done 2024-03-28T16:15:50.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.753+0000 7f7a88ff9640 1 -- 172.21.15.7:0/1766437136 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80010870 con 0x7f7a8c0b1890 2024-03-28T16:15:50.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.753+0000 7f7a88ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.753+0000 7f7a88ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:15:50.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.753+0000 7f7a88ff9640 10 monclient: dump: 2024-03-28T16:15:50.842 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.842 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.842 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.842 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.842 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.843 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.843 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.843 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.843 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.843 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.754+0000 7f7a9285c640 10 client.4323.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:50.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.754+0000 7f7a9285c640 15 client.4323.objecter _session_command_op_assign -1 1 2024-03-28T16:15:50.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.754+0000 7f7a9285c640 20 client.4323.objecter _get_session osd=-1 returning homeless 2024-03-28T16:15:50.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.754+0000 7f7a9285c640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:50.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.754+0000 7f7a9285c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:50.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.754+0000 7f7a9285c640 1 -- 172.21.15.7:0/1766437136 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7a8c142c70 con 0x7f7a8c0b1890 2024-03-28T16:15:50.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.754+0000 7f5f967fc640 10 client.4314.objecter ms_handle_connect 0x7f5f840729c0 2024-03-28T16:15:50.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.754+0000 7f5fa56c4640 1 --2- 172.21.15.7:0/3008119090 >> [v2:172.21.15.83:6826/35296,v1:172.21.15.83:6827/35296] conn(0x7f5f840729c0 0x7f5f84074de0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.758+0000 7fd302ffd640 1 -- 172.21.15.7:0/4209453085 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd30403eae0 con 0x7fd30c0a8b60 2024-03-28T16:15:50.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.758+0000 7fd302ffd640 1 --2- 172.21.15.7:0/4209453085 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fd2f006c5e0 0x7fd2f006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.760+0000 7fd311aaf640 1 --2- 172.21.15.7:0/4209453085 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fd2f006c5e0 0x7fd2f006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.760+0000 7fd311aaf640 10 monclient: get_auth_request con 0x7fd2f006c5e0 auth_method 0 2024-03-28T16:15:50.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.760+0000 7fd311aaf640 1 --2- 172.21.15.7:0/4209453085 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fd2f006c5e0 0x7fd2f006eaa0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd30c147210 tx=0x7fd2fc007990 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.760+0000 7fd302ffd640 10 client.4330.objecter ms_handle_connect 0x7fd2f006c5e0 2024-03-28T16:15:50.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.761+0000 7f34697fa640 1 -- 172.21.15.7:0/3791127862 <== osd.3 v2:172.21.15.7:6816/34921 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f3458075440 con 0x7f3458072970 2024-03-28T16:15:50.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.761+0000 7f34697fa640 10 client.4277.objecter ms_dispatch 0x7f346c000b90 command_reply(tid 1: 0 ) v1 2024-03-28T16:15:50.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.761+0000 7f34697fa640 10 client.4277.objecter _finish_command 1 = system:0 2024-03-28T16:15:50.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.761+0000 7f34697fa640 20 client.4277.objecter put_session s=0x7f34580722a0 osd=3 4 2024-03-28T16:15:50.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.761+0000 7f34697fa640 15 client.4277.objecter _session_command_op_remove 3 1 2024-03-28T16:15:50.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.762+0000 7fd302ffd640 1 -- 172.21.15.7:0/4209453085 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd30407be80 con 0x7fd30c0a8b60 2024-03-28T16:15:50.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.762+0000 7fd302ffd640 10 client.4330.objecter ms_dispatch 0x7fd30c000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:50.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.762+0000 7fd302ffd640 3 client.4330.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:50.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.762+0000 7fd302ffd640 3 client.4330.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:50.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.762+0000 7fd302ffd640 10 client.4330.objecter checking command 1 2024-03-28T16:15:50.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.762+0000 7fd302ffd640 1 --2- 172.21.15.7:0/4209453085 >> [v2:172.21.15.7:6808/34918,v1:172.21.15.7:6809/34918] conn(0x7fd2f00729c0 0x7fd2f0074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.762+0000 7fd302ffd640 20 client.4330.objecter _get_session s=0x7fd2f00722f0 osd=0 3 2024-03-28T16:15:50.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.762+0000 7fd302ffd640 20 client.4330.objecter put_session s=0x7fd2f00722f0 osd=0 3 2024-03-28T16:15:50.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.762+0000 7fd302ffd640 15 client.4330.objecter _session_command_op_remove -1 1 2024-03-28T16:15:50.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.762+0000 7fd302ffd640 20 client.4330.objecter _get_session s=0x7fd2f00722f0 osd=0 3 2024-03-28T16:15:50.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.762+0000 7fd302ffd640 20 client.4330.objecter get_session s=0x7fd2f00722f0 osd=0 3 2024-03-28T16:15:50.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.762+0000 7fd302ffd640 15 client.4330.objecter _session_command_op_assign 0 1 2024-03-28T16:15:50.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.762+0000 7fd302ffd640 20 client.4330.objecter put_session s=0x7fd2f00722f0 osd=0 4 2024-03-28T16:15:50.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.762+0000 7fd302ffd640 10 client.4330.objecter _send_command 1 2024-03-28T16:15:50.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.762+0000 7fd302ffd640 1 -- 172.21.15.7:0/4209453085 --> [v2:172.21.15.7:6808/34918,v1:172.21.15.7:6809/34918] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2f0075490 con 0x7fd2f00729c0 2024-03-28T16:15:50.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.762+0000 7fd302ffd640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-28T16:15:50.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7fd302ffd640 1 -- 172.21.15.7:0/4209453085 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd304077ac0 con 0x7fd30c0a8b60 2024-03-28T16:15:50.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7fd302ffd640 10 monclient: handle_get_version_reply finishing 1 version 28 2024-03-28T16:15:50.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7fd312ab1640 1 --2- 172.21.15.7:0/4209453085 >> [v2:172.21.15.7:6808/34918,v1:172.21.15.7:6809/34918] conn(0x7fd2f00729c0 0x7fd2f0074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f07517fa640 1 -- 172.21.15.7:0/2546658315 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074c003b70 con 0x7f075413e000 2024-03-28T16:15:50.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f07517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f07517fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:50.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f07517fa640 10 monclient: dump: 2024-03-28T16:15:50.849 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.849 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.849 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.849 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.849 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.849 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.849 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.849 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.849 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.849 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f07517fa640 10 monclient: _finish_auth 0 2024-03-28T16:15:50.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f07517fa640 10 monclient: _check_auth_tickets 2024-03-28T16:15:50.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f07517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:50.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f07517fa640 1 -- 172.21.15.7:0/2546658315 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f074c003cf0 con 0x7f075413e000 2024-03-28T16:15:50.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f07517fa640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:50.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f07517fa640 1 -- 172.21.15.7:0/2546658315 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074c0052b0 con 0x7f075413e000 2024-03-28T16:15:50.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f07517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f07517fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:50.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f07517fa640 10 monclient: dump: 2024-03-28T16:15:50.850 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.850 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.850 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.850 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.850 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.850 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.850 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.850 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.850 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.850 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f075b037640 5 monclient: authenticate success, global_id 4324 2024-03-28T16:15:50.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f075b037640 10 monclient: _renew_subs 2024-03-28T16:15:50.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f075b037640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:50.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f075b037640 1 -- 172.21.15.7:0/2546658315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0754140bd0 con 0x7f075413e000 2024-03-28T16:15:50.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f0752ffd640 4 set_mon_vals no callback set 2024-03-28T16:15:50.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f075b037640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:50.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f075b037640 10 monclient: _renew_subs 2024-03-28T16:15:50.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f075b037640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:50.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f075b037640 1 -- 172.21.15.7:0/2546658315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07541410f0 con 0x7f075413e000 2024-03-28T16:15:50.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7f075b037640 1 librados: init done 2024-03-28T16:15:50.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.763+0000 7fd312ab1640 10 monclient: get_auth_request con 0x7fd2f00729c0 auth_method 0 2024-03-28T16:15:50.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.764+0000 7f075b037640 10 client.4324.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:50.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.764+0000 7f075b037640 15 client.4324.objecter _session_command_op_assign -1 1 2024-03-28T16:15:50.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.764+0000 7f075b037640 20 client.4324.objecter _get_session osd=-1 returning homeless 2024-03-28T16:15:50.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.764+0000 7f075b037640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:50.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.764+0000 7f075b037640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:50.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.764+0000 7f075b037640 1 -- 172.21.15.7:0/2546658315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f07540b35c0 con 0x7f075413e000 2024-03-28T16:15:50.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c057fa640 1 -- 172.21.15.7:0/312124861 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc0035f0 con 0x7f6c0002ec00 2024-03-28T16:15:50.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c057fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:50.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c057fa640 10 monclient: dump: 2024-03-28T16:15:50.853 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.854 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.854 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.854 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.854 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.854 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.854 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.854 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.854 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.854 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c057fa640 10 monclient: _finish_auth 0 2024-03-28T16:15:50.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c057fa640 10 monclient: _check_auth_tickets 2024-03-28T16:15:50.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:50.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c057fa640 1 -- 172.21.15.7:0/312124861 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bfc003750 con 0x7f6c0002ec00 2024-03-28T16:15:50.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c057fa640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:50.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c057fa640 1 -- 172.21.15.7:0/312124861 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc003b00 con 0x7f6c0002ec00 2024-03-28T16:15:50.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:50.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c057fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:50.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c057fa640 10 monclient: dump: 2024-03-28T16:15:50.855 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:50.855 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:50.855 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.855 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:50.855 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:50.855 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:50.855 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:50.855 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:50.855 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:50.856 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:50.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c0ed37640 5 monclient: authenticate success, global_id 4333 2024-03-28T16:15:50.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c0ed37640 10 monclient: _renew_subs 2024-03-28T16:15:50.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c0ed37640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:50.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c0ed37640 1 -- 172.21.15.7:0/312124861 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c00029550 con 0x7f6c0002ec00 2024-03-28T16:15:50.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c06ffd640 4 set_mon_vals no callback set 2024-03-28T16:15:50.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c0ed37640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:50.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c0ed37640 10 monclient: _renew_subs 2024-03-28T16:15:50.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c0ed37640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:50.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c0ed37640 1 -- 172.21.15.7:0/312124861 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c000ce660 con 0x7f6c0002ec00 2024-03-28T16:15:50.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.766+0000 7f6c0ed37640 1 librados: init done 2024-03-28T16:15:50.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.767+0000 7f6c0ed37640 10 client.4333.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:50.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.767+0000 7f6c0ed37640 15 client.4333.objecter _session_command_op_assign -1 1 2024-03-28T16:15:50.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.767+0000 7f6c0ed37640 20 client.4333.objecter _get_session osd=-1 returning homeless 2024-03-28T16:15:50.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.767+0000 7f6c0ed37640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:50.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.767+0000 7f6c0ed37640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:50.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.767+0000 7f6c0ed37640 1 -- 172.21.15.7:0/312124861 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6c000d68b0 con 0x7f6c0002ec00 2024-03-28T16:15:50.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.768+0000 7fd302ffd640 10 client.4330.objecter ms_handle_connect 0x7fd2f00729c0 2024-03-28T16:15:50.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.768+0000 7fd312ab1640 1 --2- 172.21.15.7:0/4209453085 >> [v2:172.21.15.7:6808/34918,v1:172.21.15.7:6809/34918] conn(0x7fd2f00729c0 0x7fd2f0074de0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.785+0000 7f51c57fa640 1 -- 172.21.15.7:0/753227592 <== osd.7 v2:172.21.15.83:6802/35295 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f51ac075470 con 0x7f51ac0729a0 2024-03-28T16:15:50.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.785+0000 7f51c57fa640 10 client.4321.objecter ms_dispatch 0x7f51c8000b90 command_reply(tid 1: 0 ) v1 2024-03-28T16:15:50.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.786+0000 7f51c57fa640 10 client.4321.objecter _finish_command 1 = system:0 2024-03-28T16:15:50.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.786+0000 7f51c57fa640 20 client.4321.objecter put_session s=0x7f51ac0722d0 osd=7 4 2024-03-28T16:15:50.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.786+0000 7f51c57fa640 15 client.4321.objecter _session_command_op_remove 7 1 2024-03-28T16:15:50.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.786+0000 7f7a88ff9640 1 -- 172.21.15.7:0/1766437136 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7a80010b40 con 0x7f7a8c0b1890 2024-03-28T16:15:50.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.787+0000 7f7a88ff9640 1 --2- 172.21.15.7:0/1766437136 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f7a7806c5f0 0x7f7a7806eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.787+0000 7f7a8bfff640 1 --2- 172.21.15.7:0/1766437136 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f7a7806c5f0 0x7f7a7806eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.787+0000 7f7a8bfff640 10 monclient: get_auth_request con 0x7f7a7806c5f0 auth_method 0 2024-03-28T16:15:50.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.787+0000 7f7a8bfff640 1 --2- 172.21.15.7:0/1766437136 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f7a7806c5f0 0x7f7a7806eab0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c036000 tx=0x7f7a7c030000 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.787+0000 7f7a88ff9640 10 client.4323.objecter ms_handle_connect 0x7f7a7806c5f0 2024-03-28T16:15:50.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.788+0000 7f3473173640 10 client.4277.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-28T16:15:50.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.788+0000 7f3473173640 15 client.4277.objecter _session_command_op_assign -1 2 2024-03-28T16:15:50.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.788+0000 7f3473173640 20 client.4277.objecter _get_session s=0x7f34580722a0 osd=3 3 2024-03-28T16:15:50.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.788+0000 7f3473173640 20 client.4277.objecter put_session s=0x7f34580722a0 osd=3 3 2024-03-28T16:15:50.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.788+0000 7f3473173640 20 client.4277.objecter _get_session s=0x7f34580722a0 osd=3 3 2024-03-28T16:15:50.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.788+0000 7f3473173640 15 client.4277.objecter _session_command_op_remove -1 2 2024-03-28T16:15:50.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.788+0000 7f3473173640 20 client.4277.objecter get_session s=0x7f34580722a0 osd=3 3 2024-03-28T16:15:50.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.788+0000 7f3473173640 15 client.4277.objecter _session_command_op_assign 3 2 2024-03-28T16:15:50.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.788+0000 7f3473173640 20 client.4277.objecter put_session s=0x7f34580722a0 osd=3 4 2024-03-28T16:15:50.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.788+0000 7f3473173640 10 client.4277.objecter _send_command 2 2024-03-28T16:15:50.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.788+0000 7f3473173640 1 -- 172.21.15.7:0/3791127862 --> [v2:172.21.15.7:6816/34921,v1:172.21.15.7:6817/34921] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f346c13db00 con 0x7f3458072970 2024-03-28T16:15:50.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.796+0000 7f07517fa640 1 -- 172.21.15.7:0/2546658315 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f074c005410 con 0x7f075413e000 2024-03-28T16:15:50.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.796+0000 7f07517fa640 1 --2- 172.21.15.7:0/2546658315 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f074006c5e0 0x7f074006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.797+0000 7f0758dd3640 1 --2- 172.21.15.7:0/2546658315 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f074006c5e0 0x7f074006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.797+0000 7f0758dd3640 10 monclient: get_auth_request con 0x7f074006c5e0 auth_method 0 2024-03-28T16:15:50.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.797+0000 7f0758dd3640 1 --2- 172.21.15.7:0/2546658315 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f074006c5e0 0x7f074006eaa0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0748001610 tx=0x7f0748007930 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.797+0000 7f07517fa640 10 client.4324.objecter ms_handle_connect 0x7f074006c5e0 2024-03-28T16:15:50.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.799+0000 7f34697fa640 1 -- 172.21.15.7:0/3791127862 <== osd.3 v2:172.21.15.7:6816/34921 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f346c13db00 con 0x7f3458072970 2024-03-28T16:15:50.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.799+0000 7f34697fa640 10 client.4277.objecter ms_dispatch 0x7f346c000b90 command_reply(tid 2: 0 ) v1 2024-03-28T16:15:50.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.799+0000 7f34697fa640 10 client.4277.objecter _finish_command 2 = system:0 2024-03-28T16:15:50.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.799+0000 7f34697fa640 20 client.4277.objecter put_session s=0x7f34580722a0 osd=3 4 2024-03-28T16:15:50.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.799+0000 7f34697fa640 15 client.4277.objecter _session_command_op_remove 3 2 2024-03-28T16:15:50.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.800+0000 7f3473173640 10 librados: watch_flush enter 2024-03-28T16:15:50.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.800+0000 7f3473173640 10 librados: watch_flush exit 2024-03-28T16:15:50.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.800+0000 7f3473173640 10 client.4277.objecter close_session for osd.3 2024-03-28T16:15:50.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.800+0000 7f3473173640 1 -- 172.21.15.7:0/3791127862 >> [v2:172.21.15.7:6816/34921,v1:172.21.15.7:6817/34921] conn(0x7f3458072970 msgr2=0x7f3458074d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.800+0000 7f3473173640 1 --2- 172.21.15.7:0/3791127862 >> [v2:172.21.15.7:6816/34921,v1:172.21.15.7:6817/34921] conn(0x7f3458072970 0x7f3458074d90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.800+0000 7f3473173640 20 client.4277.objecter put_session s=0x7f34580722a0 osd=3 1 2024-03-28T16:15:50.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.800+0000 7f3473173640 20 client.4277.objecter shutdown clearing up homeless session... 2024-03-28T16:15:50.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.800+0000 7f3473173640 10 client.4277.objecter successfully canceled tick 2024-03-28T16:15:50.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.800+0000 7f3473173640 1 -- 172.21.15.7:0/3791127862 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f345806c590 msgr2=0x7f345806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.800+0000 7f3473173640 1 --2- 172.21.15.7:0/3791127862 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f345806c590 0x7f345806ea50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f345c009710 tx=0x7f345c006c90 comp rx=0 tx=0).stop 2024-03-28T16:15:50.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.800+0000 7f3473173640 10 monclient: shutdown 2024-03-28T16:15:50.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.800+0000 7f3473173640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:50.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.800+0000 7f3473173640 1 -- 172.21.15.7:0/3791127862 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f346c0ad840 msgr2=0x7f346c0b2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.800+0000 7f3473173640 1 --2- 172.21.15.7:0/3791127862 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f346c0ad840 0x7f346c0b2400 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3464009e50 tx=0x7f3464004a40 comp rx=0 tx=0).stop 2024-03-28T16:15:50.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.800+0000 7f3471710640 1 -- 172.21.15.7:0/3791127862 reap_dead start 2024-03-28T16:15:50.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.800+0000 7f3473173640 1 -- 172.21.15.7:0/3791127862 shutdown_connections 2024-03-28T16:15:50.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.800+0000 7f3473173640 1 -- 172.21.15.7:0/3791127862 >> 172.21.15.7:0/3791127862 conn(0x7f346c0b2db0 msgr2=0x7f346c138550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:50.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.800+0000 7f3473173640 1 -- 172.21.15.7:0/3791127862 shutdown_connections 2024-03-28T16:15:50.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.800+0000 7f3473173640 1 -- 172.21.15.7:0/3791127862 wait complete. 2024-03-28T16:15:50.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.800+0000 7f3473173640 1 librados: shutdown 2024-03-28T16:15:50.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.802+0000 7f51cdc08640 10 client.4321.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-28T16:15:50.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.802+0000 7f51cdc08640 15 client.4321.objecter _session_command_op_assign -1 2 2024-03-28T16:15:50.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.802+0000 7f51cdc08640 20 client.4321.objecter _get_session s=0x7f51ac0722d0 osd=7 3 2024-03-28T16:15:50.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.802+0000 7f51cdc08640 20 client.4321.objecter put_session s=0x7f51ac0722d0 osd=7 3 2024-03-28T16:15:50.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.802+0000 7f51cdc08640 20 client.4321.objecter _get_session s=0x7f51ac0722d0 osd=7 3 2024-03-28T16:15:50.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.802+0000 7f51cdc08640 15 client.4321.objecter _session_command_op_remove -1 2 2024-03-28T16:15:50.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.802+0000 7f51cdc08640 20 client.4321.objecter get_session s=0x7f51ac0722d0 osd=7 3 2024-03-28T16:15:50.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.802+0000 7f51cdc08640 15 client.4321.objecter _session_command_op_assign 7 2 2024-03-28T16:15:50.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.802+0000 7f51cdc08640 20 client.4321.objecter put_session s=0x7f51ac0722d0 osd=7 4 2024-03-28T16:15:50.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.802+0000 7f51cdc08640 10 client.4321.objecter _send_command 2 2024-03-28T16:15:50.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.802+0000 7f51cdc08640 1 -- 172.21.15.7:0/753227592 --> [v2:172.21.15.83:6802/35295,v1:172.21.15.83:6803/35295] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f51c80b3700 con 0x7f51ac0729a0 2024-03-28T16:15:50.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.803+0000 7f07517fa640 1 -- 172.21.15.7:0/2546658315 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f074c0581d0 con 0x7f075413e000 2024-03-28T16:15:50.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.803+0000 7f07517fa640 10 client.4324.objecter ms_dispatch 0x7f0754000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:50.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.803+0000 7f07517fa640 3 client.4324.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:50.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.803+0000 7f07517fa640 3 client.4324.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:50.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.803+0000 7f07517fa640 10 client.4324.objecter checking command 1 2024-03-28T16:15:50.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.803+0000 7f07517fa640 1 --2- 172.21.15.7:0/2546658315 >> [v2:172.21.15.7:6800/34919,v1:172.21.15.7:6801/34919] conn(0x7f07400729c0 0x7f0740074de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.803+0000 7f07517fa640 20 client.4324.objecter _get_session s=0x7f07400722f0 osd=1 3 2024-03-28T16:15:50.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.803+0000 7f07517fa640 20 client.4324.objecter put_session s=0x7f07400722f0 osd=1 3 2024-03-28T16:15:50.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.803+0000 7f07517fa640 15 client.4324.objecter _session_command_op_remove -1 1 2024-03-28T16:15:50.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.803+0000 7f07517fa640 20 client.4324.objecter _get_session s=0x7f07400722f0 osd=1 3 2024-03-28T16:15:50.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.803+0000 7f07517fa640 20 client.4324.objecter get_session s=0x7f07400722f0 osd=1 3 2024-03-28T16:15:50.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.803+0000 7f07517fa640 15 client.4324.objecter _session_command_op_assign 1 1 2024-03-28T16:15:50.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.803+0000 7f07517fa640 20 client.4324.objecter put_session s=0x7f07400722f0 osd=1 4 2024-03-28T16:15:50.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.803+0000 7f07517fa640 10 client.4324.objecter _send_command 1 2024-03-28T16:15:50.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.803+0000 7f07517fa640 1 -- 172.21.15.7:0/2546658315 --> [v2:172.21.15.7:6800/34919,v1:172.21.15.7:6801/34919] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0740075490 con 0x7f07400729c0 2024-03-28T16:15:50.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.803+0000 7f07517fa640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-28T16:15:50.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.803+0000 7f07517fa640 1 -- 172.21.15.7:0/2546658315 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f074c054510 con 0x7f075413e000 2024-03-28T16:15:50.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.803+0000 7f07517fa640 10 monclient: handle_get_version_reply finishing 1 version 28 2024-03-28T16:15:50.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.804+0000 7f0753fff640 1 --2- 172.21.15.7:0/2546658315 >> [v2:172.21.15.7:6800/34919,v1:172.21.15.7:6801/34919] conn(0x7f07400729c0 0x7f0740074de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.805+0000 7f0753fff640 10 monclient: get_auth_request con 0x7f07400729c0 auth_method 0 2024-03-28T16:15:50.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.805+0000 7febb9ffb640 1 -- 172.21.15.7:0/3386136446 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7febbc003970 con 0x7febc41d5a40 2024-03-28T16:15:50.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.805+0000 7febb9ffb640 1 --2- 172.21.15.7:0/3386136446 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7feba806c5f0 0x7feba806eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.806+0000 7febc9620640 1 --2- 172.21.15.7:0/3386136446 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7feba806c5f0 0x7feba806eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.806+0000 7febc9620640 10 monclient: get_auth_request con 0x7feba806c5f0 auth_method 0 2024-03-28T16:15:50.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.806+0000 7febb9ffb640 10 client.4326.objecter ms_handle_connect 0x7feba806c5f0 2024-03-28T16:15:50.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.806+0000 7febc9620640 1 --2- 172.21.15.7:0/3386136446 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7feba806c5f0 0x7feba806eab0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7febc0035000 tx=0x7febc0031000 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.809+0000 7f07517fa640 10 client.4324.objecter ms_handle_connect 0x7f07400729c0 2024-03-28T16:15:50.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.809+0000 7f0753fff640 1 --2- 172.21.15.7:0/2546658315 >> [v2:172.21.15.7:6800/34919,v1:172.21.15.7:6801/34919] conn(0x7f07400729c0 0x7f0740074de0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.869 INFO:teuthology.orchestra.run.smithi007.stdout:73014444040 2024-03-28T16:15:50.870 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-28T16:15:50.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.812+0000 7febb9ffb640 1 -- 172.21.15.7:0/3386136446 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7febbc057a40 con 0x7febc41d5a40 2024-03-28T16:15:50.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.812+0000 7febb9ffb640 10 client.4326.objecter ms_dispatch 0x7febc4000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:50.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.812+0000 7febb9ffb640 3 client.4326.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:50.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.812+0000 7febb9ffb640 3 client.4326.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:50.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.813+0000 7febb9ffb640 10 client.4326.objecter checking command 1 2024-03-28T16:15:50.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.813+0000 7febb9ffb640 1 --2- 172.21.15.7:0/3386136446 >> [v2:172.21.15.83:6818/35294,v1:172.21.15.83:6819/35294] conn(0x7feba80729d0 0x7feba8074df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.813+0000 7febb9ffb640 20 client.4326.objecter _get_session s=0x7feba8072300 osd=4 3 2024-03-28T16:15:50.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.813+0000 7febb9ffb640 20 client.4326.objecter put_session s=0x7feba8072300 osd=4 3 2024-03-28T16:15:50.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.813+0000 7febb9ffb640 15 client.4326.objecter _session_command_op_remove -1 1 2024-03-28T16:15:50.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.813+0000 7febb9ffb640 20 client.4326.objecter _get_session s=0x7feba8072300 osd=4 3 2024-03-28T16:15:50.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.813+0000 7febb9ffb640 20 client.4326.objecter get_session s=0x7feba8072300 osd=4 3 2024-03-28T16:15:50.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.813+0000 7febb9ffb640 15 client.4326.objecter _session_command_op_assign 4 1 2024-03-28T16:15:50.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.813+0000 7febb9ffb640 20 client.4326.objecter put_session s=0x7feba8072300 osd=4 4 2024-03-28T16:15:50.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.813+0000 7febb9ffb640 10 client.4326.objecter _send_command 1 2024-03-28T16:15:50.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.813+0000 7febb9ffb640 1 -- 172.21.15.7:0/3386136446 --> [v2:172.21.15.83:6818/35294,v1:172.21.15.83:6819/35294] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feba80754a0 con 0x7feba80729d0 2024-03-28T16:15:50.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.813+0000 7febb9ffb640 20 client.4326.objecter dump_active .. 0 homeless 2024-03-28T16:15:50.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.813+0000 7febb9ffb640 1 -- 172.21.15.7:0/3386136446 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7febbc053680 con 0x7febc41d5a40 2024-03-28T16:15:50.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.813+0000 7febb9ffb640 10 monclient: handle_get_version_reply finishing 1 version 28 2024-03-28T16:15:50.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.813+0000 7febc8e1f640 1 --2- 172.21.15.7:0/3386136446 >> [v2:172.21.15.83:6818/35294,v1:172.21.15.83:6819/35294] conn(0x7feba80729d0 0x7feba8074df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.813+0000 7f7a88ff9640 1 -- 172.21.15.7:0/1766437136 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7a80058b90 con 0x7f7a8c0b1890 2024-03-28T16:15:50.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.813+0000 7f7a88ff9640 10 client.4323.objecter ms_dispatch 0x7f7a8c000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:50.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.813+0000 7f7a88ff9640 3 client.4323.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:50.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.813+0000 7f7a88ff9640 3 client.4323.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:50.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.813+0000 7f7a88ff9640 10 client.4323.objecter checking command 1 2024-03-28T16:15:50.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.814+0000 7f7a88ff9640 1 --2- 172.21.15.7:0/1766437136 >> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] conn(0x7f7a780729d0 0x7f7a78074df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.814+0000 7f7a88ff9640 20 client.4323.objecter _get_session s=0x7f7a78072300 osd=6 3 2024-03-28T16:15:50.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.814+0000 7f7a88ff9640 20 client.4323.objecter put_session s=0x7f7a78072300 osd=6 3 2024-03-28T16:15:50.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.814+0000 7f7a88ff9640 15 client.4323.objecter _session_command_op_remove -1 1 2024-03-28T16:15:50.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.814+0000 7f7a88ff9640 20 client.4323.objecter _get_session s=0x7f7a78072300 osd=6 3 2024-03-28T16:15:50.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.814+0000 7f7a88ff9640 20 client.4323.objecter get_session s=0x7f7a78072300 osd=6 3 2024-03-28T16:15:50.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.814+0000 7f7a88ff9640 15 client.4323.objecter _session_command_op_assign 6 1 2024-03-28T16:15:50.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.814+0000 7f7a88ff9640 20 client.4323.objecter put_session s=0x7f7a78072300 osd=6 4 2024-03-28T16:15:50.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.814+0000 7f7a88ff9640 10 client.4323.objecter _send_command 1 2024-03-28T16:15:50.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.814+0000 7f7a88ff9640 1 -- 172.21.15.7:0/1766437136 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a780754a0 con 0x7f7a780729d0 2024-03-28T16:15:50.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.814+0000 7f7a88ff9640 20 client.4323.objecter dump_active .. 0 homeless 2024-03-28T16:15:50.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.814+0000 7f7a88ff9640 1 -- 172.21.15.7:0/1766437136 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a800547d0 con 0x7f7a8c0b1890 2024-03-28T16:15:50.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.814+0000 7f7a88ff9640 10 monclient: handle_get_version_reply finishing 1 version 28 2024-03-28T16:15:50.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.814+0000 7febc8e1f640 10 monclient: get_auth_request con 0x7feba80729d0 auth_method 0 2024-03-28T16:15:50.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.814+0000 7f7a90df9640 1 --2- 172.21.15.7:0/1766437136 >> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] conn(0x7f7a780729d0 0x7f7a78074df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.815+0000 7f7a90df9640 10 monclient: get_auth_request con 0x7f7a780729d0 auth_method 0 2024-03-28T16:15:50.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.817+0000 7febb9ffb640 10 client.4326.objecter ms_handle_connect 0x7feba80729d0 2024-03-28T16:15:50.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.817+0000 7febc8e1f640 1 --2- 172.21.15.7:0/3386136446 >> [v2:172.21.15.83:6818/35294,v1:172.21.15.83:6819/35294] conn(0x7feba80729d0 0x7feba8074df0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.819+0000 7f7a88ff9640 10 client.4323.objecter ms_handle_connect 0x7f7a780729d0 2024-03-28T16:15:50.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.819+0000 7f7a90df9640 1 --2- 172.21.15.7:0/1766437136 >> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] conn(0x7f7a780729d0 0x7f7a78074df0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.823+0000 7f51c57fa640 1 -- 172.21.15.7:0/753227592 <== osd.7 v2:172.21.15.83:6802/35295 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f51c80b3700 con 0x7f51ac0729a0 2024-03-28T16:15:50.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.823+0000 7f51c57fa640 10 client.4321.objecter ms_dispatch 0x7f51c8000b90 command_reply(tid 2: 0 ) v1 2024-03-28T16:15:50.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.823+0000 7f51c57fa640 10 client.4321.objecter _finish_command 2 = system:0 2024-03-28T16:15:50.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.823+0000 7f51c57fa640 20 client.4321.objecter put_session s=0x7f51ac0722d0 osd=7 4 2024-03-28T16:15:50.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.823+0000 7f51c57fa640 15 client.4321.objecter _session_command_op_remove 7 2 2024-03-28T16:15:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.824+0000 7f51cdc08640 10 librados: watch_flush enter 2024-03-28T16:15:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.824+0000 7f51cdc08640 10 librados: watch_flush exit 2024-03-28T16:15:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.824+0000 7f51cdc08640 10 client.4321.objecter close_session for osd.7 2024-03-28T16:15:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.824+0000 7f51cdc08640 1 -- 172.21.15.7:0/753227592 >> [v2:172.21.15.83:6802/35295,v1:172.21.15.83:6803/35295] conn(0x7f51ac0729a0 msgr2=0x7f51ac074dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.824+0000 7f51cdc08640 1 --2- 172.21.15.7:0/753227592 >> [v2:172.21.15.83:6802/35295,v1:172.21.15.83:6803/35295] conn(0x7f51ac0729a0 0x7f51ac074dc0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.824+0000 7f51cdc08640 20 client.4321.objecter put_session s=0x7f51ac0722d0 osd=7 1 2024-03-28T16:15:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.824+0000 7f51cdc08640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-28T16:15:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.824+0000 7f51cdc08640 10 client.4321.objecter successfully canceled tick 2024-03-28T16:15:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.824+0000 7f51cdc08640 1 -- 172.21.15.7:0/753227592 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f51ac06c5c0 msgr2=0x7f51ac06ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.824+0000 7f51cdc08640 1 --2- 172.21.15.7:0/753227592 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f51ac06c5c0 0x7f51ac06ea80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f51c81443b0 tx=0x7f51b4006c90 comp rx=0 tx=0).stop 2024-03-28T16:15:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.824+0000 7f51cdc08640 10 monclient: shutdown 2024-03-28T16:15:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.824+0000 7f51cdc08640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.824+0000 7f51cdc08640 1 -- 172.21.15.7:0/753227592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51c80b32e0 msgr2=0x7f51c8143710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.824+0000 7f51cdc08640 1 --2- 172.21.15.7:0/753227592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51c80b32e0 0x7f51c8143710 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f51bc00c820 tx=0x7f51bc00cce0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.824+0000 7f51cd407640 1 -- 172.21.15.7:0/753227592 reap_dead start 2024-03-28T16:15:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.824+0000 7f51cdc08640 1 -- 172.21.15.7:0/753227592 shutdown_connections 2024-03-28T16:15:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.824+0000 7f51cdc08640 1 -- 172.21.15.7:0/753227592 >> 172.21.15.7:0/753227592 conn(0x7f51c80711a0 msgr2=0x7f51c80b6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.824+0000 7f51cdc08640 1 -- 172.21.15.7:0/753227592 shutdown_connections 2024-03-28T16:15:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.824+0000 7f51cdc08640 1 -- 172.21.15.7:0/753227592 wait complete. 2024-03-28T16:15:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.824+0000 7f51cdc08640 1 librados: shutdown 2024-03-28T16:15:50.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.832+0000 7f5f967fc640 1 -- 172.21.15.7:0/3008119090 <== osd.5 v2:172.21.15.83:6826/35296 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f5f84075490 con 0x7f5f840729c0 2024-03-28T16:15:50.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.832+0000 7f5f967fc640 10 client.4314.objecter ms_dispatch 0x7f5fa0000b90 command_reply(tid 1: 0 ) v1 2024-03-28T16:15:50.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.832+0000 7f5f967fc640 10 client.4314.objecter _finish_command 1 = system:0 2024-03-28T16:15:50.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.832+0000 7f5f967fc640 20 client.4314.objecter put_session s=0x7f5f840722f0 osd=5 4 2024-03-28T16:15:50.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.832+0000 7f5f967fc640 15 client.4314.objecter _session_command_op_remove 5 1 2024-03-28T16:15:50.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.834+0000 7fd302ffd640 1 -- 172.21.15.7:0/4209453085 <== osd.0 v2:172.21.15.7:6808/34918 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fd2f0075490 con 0x7fd2f00729c0 2024-03-28T16:15:50.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.834+0000 7fd302ffd640 10 client.4330.objecter ms_dispatch 0x7fd30c000b90 command_reply(tid 1: 0 ) v1 2024-03-28T16:15:50.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.834+0000 7fd302ffd640 10 client.4330.objecter _finish_command 1 = system:0 2024-03-28T16:15:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.834+0000 7fd302ffd640 20 client.4330.objecter put_session s=0x7fd2f00722f0 osd=0 4 2024-03-28T16:15:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.834+0000 7fd302ffd640 15 client.4330.objecter _session_command_op_remove 0 1 2024-03-28T16:15:50.881 INFO:teuthology.orchestra.run.smithi007.stdout:64424509447 2024-03-28T16:15:50.882 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-28T16:15:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.840+0000 7f6c057fa640 1 -- 172.21.15.7:0/312124861 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6bfc003c60 con 0x7f6c0002ec00 2024-03-28T16:15:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.840+0000 7f6c057fa640 1 --2- 172.21.15.7:0/312124861 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f6bec06c5c0 0x7f6bec06ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.841+0000 7f6c0cad3640 1 --2- 172.21.15.7:0/312124861 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f6bec06c5c0 0x7f6bec06ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.841+0000 7f6c0cad3640 10 monclient: get_auth_request con 0x7f6bec06c5c0 auth_method 0 2024-03-28T16:15:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.841+0000 7f6c0cad3640 1 --2- 172.21.15.7:0/312124861 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f6bec06c5c0 0x7f6bec06ea80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6c08000b60 tx=0x7f6c0806c040 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.841+0000 7f6c057fa640 10 client.4333.objecter ms_handle_connect 0x7f6bec06c5c0 2024-03-28T16:15:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.845+0000 7f6c057fa640 1 -- 172.21.15.7:0/312124861 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6bfc057dc0 con 0x7f6c0002ec00 2024-03-28T16:15:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.845+0000 7f6c057fa640 10 client.4333.objecter ms_dispatch 0x7f6c08000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.845+0000 7f6c057fa640 3 client.4333.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.845+0000 7f6c057fa640 3 client.4333.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.845+0000 7f6c057fa640 10 client.4333.objecter checking command 1 2024-03-28T16:15:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.845+0000 7f6c057fa640 1 --2- 172.21.15.7:0/312124861 >> [v2:172.21.15.7:6824/34920,v1:172.21.15.7:6825/34920] conn(0x7f6bec0729a0 0x7f6bec074dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.845+0000 7f6c057fa640 20 client.4333.objecter _get_session s=0x7f6bec0722d0 osd=2 3 2024-03-28T16:15:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.845+0000 7f6c057fa640 20 client.4333.objecter put_session s=0x7f6bec0722d0 osd=2 3 2024-03-28T16:15:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.845+0000 7f6c057fa640 15 client.4333.objecter _session_command_op_remove -1 1 2024-03-28T16:15:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.845+0000 7f6c057fa640 20 client.4333.objecter _get_session s=0x7f6bec0722d0 osd=2 3 2024-03-28T16:15:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.845+0000 7f6c057fa640 20 client.4333.objecter get_session s=0x7f6bec0722d0 osd=2 3 2024-03-28T16:15:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.845+0000 7f6c057fa640 15 client.4333.objecter _session_command_op_assign 2 1 2024-03-28T16:15:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.845+0000 7f6c057fa640 20 client.4333.objecter put_session s=0x7f6bec0722d0 osd=2 4 2024-03-28T16:15:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.845+0000 7f6c057fa640 10 client.4333.objecter _send_command 1 2024-03-28T16:15:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.845+0000 7f6c057fa640 1 -- 172.21.15.7:0/312124861 --> [v2:172.21.15.7:6824/34920,v1:172.21.15.7:6825/34920] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6bec075470 con 0x7f6bec0729a0 2024-03-28T16:15:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.845+0000 7f6c057fa640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-28T16:15:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.846+0000 7f6c0d2d4640 1 --2- 172.21.15.7:0/312124861 >> [v2:172.21.15.7:6824/34920,v1:172.21.15.7:6825/34920] conn(0x7f6bec0729a0 0x7f6bec074dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.846+0000 7f6c0d2d4640 10 monclient: get_auth_request con 0x7f6bec0729a0 auth_method 0 2024-03-28T16:15:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.850+0000 7f6c057fa640 10 client.4333.objecter ms_handle_connect 0x7f6bec0729a0 2024-03-28T16:15:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.850+0000 7f6c0d2d4640 1 --2- 172.21.15.7:0/312124861 >> [v2:172.21.15.7:6824/34920,v1:172.21.15.7:6825/34920] conn(0x7f6bec0729a0 0x7f6bec074dc0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.862+0000 7f5fa6ec7640 10 client.4314.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-28T16:15:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.862+0000 7f5fa6ec7640 15 client.4314.objecter _session_command_op_assign -1 2 2024-03-28T16:15:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.862+0000 7f5fa6ec7640 20 client.4314.objecter _get_session s=0x7f5f840722f0 osd=5 3 2024-03-28T16:15:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.862+0000 7f5fa6ec7640 20 client.4314.objecter put_session s=0x7f5f840722f0 osd=5 3 2024-03-28T16:15:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.862+0000 7f5fa6ec7640 20 client.4314.objecter _get_session s=0x7f5f840722f0 osd=5 3 2024-03-28T16:15:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.863+0000 7f5fa6ec7640 15 client.4314.objecter _session_command_op_remove -1 2 2024-03-28T16:15:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.863+0000 7f5fa6ec7640 20 client.4314.objecter get_session s=0x7f5f840722f0 osd=5 3 2024-03-28T16:15:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.863+0000 7f5fa6ec7640 15 client.4314.objecter _session_command_op_assign 5 2 2024-03-28T16:15:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.863+0000 7f5fa6ec7640 20 client.4314.objecter put_session s=0x7f5f840722f0 osd=5 4 2024-03-28T16:15:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.863+0000 7f5fa6ec7640 10 client.4314.objecter _send_command 2 2024-03-28T16:15:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.863+0000 7f5fa6ec7640 1 -- 172.21.15.7:0/3008119090 --> [v2:172.21.15.83:6826/35296,v1:172.21.15.83:6827/35296] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5fa00af0f0 con 0x7f5f840729c0 2024-03-28T16:15:50.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.864+0000 7fd314514640 10 client.4330.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-28T16:15:50.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.864+0000 7fd314514640 15 client.4330.objecter _session_command_op_assign -1 2 2024-03-28T16:15:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.864+0000 7fd314514640 20 client.4330.objecter _get_session s=0x7fd2f00722f0 osd=0 3 2024-03-28T16:15:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.864+0000 7fd314514640 20 client.4330.objecter put_session s=0x7fd2f00722f0 osd=0 3 2024-03-28T16:15:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.864+0000 7fd314514640 20 client.4330.objecter _get_session s=0x7fd2f00722f0 osd=0 3 2024-03-28T16:15:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.864+0000 7fd314514640 15 client.4330.objecter _session_command_op_remove -1 2 2024-03-28T16:15:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.864+0000 7fd314514640 20 client.4330.objecter get_session s=0x7fd2f00722f0 osd=0 3 2024-03-28T16:15:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.864+0000 7fd314514640 15 client.4330.objecter _session_command_op_assign 0 2 2024-03-28T16:15:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.864+0000 7fd314514640 20 client.4330.objecter put_session s=0x7fd2f00722f0 osd=0 4 2024-03-28T16:15:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.864+0000 7fd314514640 10 client.4330.objecter _send_command 2 2024-03-28T16:15:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.864+0000 7fd314514640 1 -- 172.21.15.7:0/4209453085 --> [v2:172.21.15.7:6808/34918,v1:172.21.15.7:6809/34918] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd2f40011c0 con 0x7fd2f00729c0 2024-03-28T16:15:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.870+0000 7fd302ffd640 1 -- 172.21.15.7:0/4209453085 <== osd.0 v2:172.21.15.7:6808/34918 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd2f40011c0 con 0x7fd2f00729c0 2024-03-28T16:15:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.870+0000 7fd302ffd640 10 client.4330.objecter ms_dispatch 0x7fd30c000b90 command_reply(tid 2: 0 ) v1 2024-03-28T16:15:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.870+0000 7fd302ffd640 10 client.4330.objecter _finish_command 2 = system:0 2024-03-28T16:15:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.870+0000 7fd302ffd640 20 client.4330.objecter put_session s=0x7fd2f00722f0 osd=0 4 2024-03-28T16:15:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.870+0000 7fd302ffd640 15 client.4330.objecter _session_command_op_remove 0 2 2024-03-28T16:15:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.870+0000 7febb9ffb640 1 -- 172.21.15.7:0/3386136446 <== osd.4 v2:172.21.15.83:6818/35294 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7feba80754a0 con 0x7feba80729d0 2024-03-28T16:15:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.870+0000 7febb9ffb640 10 client.4326.objecter ms_dispatch 0x7febc4000b90 command_reply(tid 1: 0 ) v1 2024-03-28T16:15:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.870+0000 7febb9ffb640 10 client.4326.objecter _finish_command 1 = system:0 2024-03-28T16:15:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.870+0000 7febb9ffb640 20 client.4326.objecter put_session s=0x7feba8072300 osd=4 4 2024-03-28T16:15:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.870+0000 7febb9ffb640 15 client.4326.objecter _session_command_op_remove 4 1 2024-03-28T16:15:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.870+0000 7fd314514640 10 librados: watch_flush enter 2024-03-28T16:15:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.870+0000 7fd314514640 10 librados: watch_flush exit 2024-03-28T16:15:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.870+0000 7fd314514640 10 client.4330.objecter close_session for osd.0 2024-03-28T16:15:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.870+0000 7fd314514640 1 -- 172.21.15.7:0/4209453085 >> [v2:172.21.15.7:6808/34918,v1:172.21.15.7:6809/34918] conn(0x7fd2f00729c0 msgr2=0x7fd2f0074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.870+0000 7fd314514640 1 --2- 172.21.15.7:0/4209453085 >> [v2:172.21.15.7:6808/34918,v1:172.21.15.7:6809/34918] conn(0x7fd2f00729c0 0x7fd2f0074de0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.870+0000 7fd314514640 20 client.4330.objecter put_session s=0x7fd2f00722f0 osd=0 1 2024-03-28T16:15:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.870+0000 7fd314514640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-28T16:15:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.870+0000 7fd314514640 10 client.4330.objecter successfully canceled tick 2024-03-28T16:15:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.871+0000 7fd314514640 1 -- 172.21.15.7:0/4209453085 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fd2f006c5e0 msgr2=0x7fd2f006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.871+0000 7fd314514640 1 --2- 172.21.15.7:0/4209453085 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fd2f006c5e0 0x7fd2f006eaa0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd30c147210 tx=0x7fd2fc007990 comp rx=0 tx=0).stop 2024-03-28T16:15:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.871+0000 7fd314514640 10 monclient: shutdown 2024-03-28T16:15:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.871+0000 7fd314514640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.871+0000 7fd314514640 1 -- 172.21.15.7:0/4209453085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd30c0a8b60 msgr2=0x7fd30c12d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.871+0000 7fd314514640 1 --2- 172.21.15.7:0/4209453085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd30c0a8b60 0x7fd30c12d7d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd3040093e0 tx=0x7fd304004a40 comp rx=0 tx=0).stop 2024-03-28T16:15:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.871+0000 7fd312ab1640 1 -- 172.21.15.7:0/4209453085 reap_dead start 2024-03-28T16:15:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.871+0000 7fd314514640 1 -- 172.21.15.7:0/4209453085 shutdown_connections 2024-03-28T16:15:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.871+0000 7fd314514640 1 -- 172.21.15.7:0/4209453085 >> 172.21.15.7:0/4209453085 conn(0x7fd30c0b2cb0 msgr2=0x7fd30c1504b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.872+0000 7fd314514640 1 -- 172.21.15.7:0/4209453085 shutdown_connections 2024-03-28T16:15:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.872+0000 7fd314514640 1 -- 172.21.15.7:0/4209453085 wait complete. 2024-03-28T16:15:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.872+0000 7fd314514640 1 librados: shutdown 2024-03-28T16:15:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.877+0000 7f7a88ff9640 1 -- 172.21.15.7:0/1766437136 <== osd.6 v2:172.21.15.83:6810/35297 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7a780754a0 con 0x7f7a780729d0 2024-03-28T16:15:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.877+0000 7f7a88ff9640 10 client.4323.objecter ms_dispatch 0x7f7a8c000b90 command_reply(tid 1: 0 ) v1 2024-03-28T16:15:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.877+0000 7f7a88ff9640 10 client.4323.objecter _finish_command 1 = system:0 2024-03-28T16:15:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.877+0000 7f7a88ff9640 20 client.4323.objecter put_session s=0x7f7a78072300 osd=6 4 2024-03-28T16:15:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.877+0000 7f7a88ff9640 15 client.4323.objecter _session_command_op_remove 6 1 2024-03-28T16:15:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.879+0000 7f07517fa640 1 -- 172.21.15.7:0/2546658315 <== osd.1 v2:172.21.15.7:6800/34919 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f0740075490 con 0x7f07400729c0 2024-03-28T16:15:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.879+0000 7f07517fa640 10 client.4324.objecter ms_dispatch 0x7f0754000b90 command_reply(tid 1: 0 ) v1 2024-03-28T16:15:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.879+0000 7f07517fa640 10 client.4324.objecter _finish_command 1 = system:0 2024-03-28T16:15:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.879+0000 7f07517fa640 20 client.4324.objecter put_session s=0x7f07400722f0 osd=1 4 2024-03-28T16:15:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.879+0000 7f07517fa640 15 client.4324.objecter _session_command_op_remove 1 1 2024-03-28T16:15:50.894 INFO:teuthology.orchestra.run.smithi007.stdout:64424509446 2024-03-28T16:15:50.895 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-28T16:15:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.888+0000 7f5f967fc640 1 -- 172.21.15.7:0/3008119090 <== osd.5 v2:172.21.15.83:6826/35296 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5f90009920 con 0x7f5f840729c0 2024-03-28T16:15:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.888+0000 7f5f967fc640 10 client.4314.objecter ms_dispatch 0x7f5fa0000b90 command_reply(tid 2: 0 ) v1 2024-03-28T16:15:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.888+0000 7f5f967fc640 10 client.4314.objecter _finish_command 2 = system:0 2024-03-28T16:15:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.888+0000 7f5f967fc640 20 client.4314.objecter put_session s=0x7f5f840722f0 osd=5 4 2024-03-28T16:15:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.888+0000 7f5f967fc640 15 client.4314.objecter _session_command_op_remove 5 2 2024-03-28T16:15:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.888+0000 7f5fa6ec7640 10 librados: watch_flush enter 2024-03-28T16:15:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.889+0000 7febca622640 10 client.4326.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-28T16:15:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.889+0000 7febca622640 15 client.4326.objecter _session_command_op_assign -1 2 2024-03-28T16:15:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.889+0000 7febca622640 20 client.4326.objecter _get_session s=0x7feba8072300 osd=4 3 2024-03-28T16:15:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.889+0000 7febca622640 20 client.4326.objecter put_session s=0x7feba8072300 osd=4 3 2024-03-28T16:15:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.889+0000 7febca622640 20 client.4326.objecter _get_session s=0x7feba8072300 osd=4 3 2024-03-28T16:15:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.889+0000 7febca622640 15 client.4326.objecter _session_command_op_remove -1 2 2024-03-28T16:15:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.889+0000 7febca622640 20 client.4326.objecter get_session s=0x7feba8072300 osd=4 3 2024-03-28T16:15:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.889+0000 7febca622640 15 client.4326.objecter _session_command_op_assign 4 2 2024-03-28T16:15:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.889+0000 7febca622640 20 client.4326.objecter put_session s=0x7feba8072300 osd=4 4 2024-03-28T16:15:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.889+0000 7febca622640 10 client.4326.objecter _send_command 2 2024-03-28T16:15:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.889+0000 7febca622640 1 -- 172.21.15.7:0/3386136446 --> [v2:172.21.15.83:6818/35294,v1:172.21.15.83:6819/35294] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7febc4069430 con 0x7feba80729d0 2024-03-28T16:15:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.890+0000 7f5fa6ec7640 10 librados: watch_flush exit 2024-03-28T16:15:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.890+0000 7f5fa6ec7640 10 client.4314.objecter close_session for osd.5 2024-03-28T16:15:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.890+0000 7f5fa6ec7640 1 -- 172.21.15.7:0/3008119090 >> [v2:172.21.15.83:6826/35296,v1:172.21.15.83:6827/35296] conn(0x7f5f840729c0 msgr2=0x7f5f84074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.890+0000 7f5fa6ec7640 1 --2- 172.21.15.7:0/3008119090 >> [v2:172.21.15.83:6826/35296,v1:172.21.15.83:6827/35296] conn(0x7f5f840729c0 0x7f5f84074de0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.891+0000 7f5fa6ec7640 20 client.4314.objecter put_session s=0x7f5f840722f0 osd=5 1 2024-03-28T16:15:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.891+0000 7f5fa6ec7640 20 client.4314.objecter shutdown clearing up homeless session... 2024-03-28T16:15:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.891+0000 7f5fa6ec7640 10 client.4314.objecter successfully canceled tick 2024-03-28T16:15:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.891+0000 7f5fa6ec7640 1 -- 172.21.15.7:0/3008119090 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f5f8406c5e0 msgr2=0x7f5f8406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.891+0000 7f5fa6ec7640 1 --2- 172.21.15.7:0/3008119090 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f5f8406c5e0 0x7f5f8406eaa0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c034000 tx=0x7f5f9c02f000 comp rx=0 tx=0).stop 2024-03-28T16:15:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.891+0000 7f5fa6ec7640 10 monclient: shutdown 2024-03-28T16:15:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.891+0000 7f5fa6ec7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.891+0000 7f5fa6ec7640 1 -- 172.21.15.7:0/3008119090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fa0149540 msgr2=0x7f5fa0140700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.891+0000 7f5fa6ec7640 1 --2- 172.21.15.7:0/3008119090 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5fa0149540 0x7f5fa0140700 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5f9800b990 tx=0x7f5f9800be50 comp rx=0 tx=0).stop 2024-03-28T16:15:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.891+0000 7f5fa66c6640 1 -- 172.21.15.7:0/3008119090 reap_dead start 2024-03-28T16:15:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.891+0000 7f5fa6ec7640 1 -- 172.21.15.7:0/3008119090 shutdown_connections 2024-03-28T16:15:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.891+0000 7f5fa6ec7640 1 -- 172.21.15.7:0/3008119090 >> 172.21.15.7:0/3008119090 conn(0x7f5fa00b1ae0 msgr2=0x7f5fa00b1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.891+0000 7f5fa6ec7640 1 -- 172.21.15.7:0/3008119090 shutdown_connections 2024-03-28T16:15:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.891+0000 7f5fa6ec7640 1 -- 172.21.15.7:0/3008119090 wait complete. 2024-03-28T16:15:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.891+0000 7f5fa6ec7640 1 librados: shutdown 2024-03-28T16:15:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.895+0000 7febb9ffb640 1 -- 172.21.15.7:0/3386136446 <== osd.4 v2:172.21.15.83:6818/35294 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7febc4069430 con 0x7feba80729d0 2024-03-28T16:15:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.895+0000 7febb9ffb640 10 client.4326.objecter ms_dispatch 0x7febc4000b90 command_reply(tid 2: 0 ) v1 2024-03-28T16:15:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.895+0000 7febb9ffb640 10 client.4326.objecter _finish_command 2 = system:0 2024-03-28T16:15:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.895+0000 7febb9ffb640 20 client.4326.objecter put_session s=0x7feba8072300 osd=4 4 2024-03-28T16:15:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.895+0000 7febb9ffb640 15 client.4326.objecter _session_command_op_remove 4 2 2024-03-28T16:15:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.896+0000 7febca622640 10 librados: watch_flush enter 2024-03-28T16:15:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.896+0000 7febca622640 10 librados: watch_flush exit 2024-03-28T16:15:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.896+0000 7febca622640 10 client.4326.objecter close_session for osd.4 2024-03-28T16:15:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.896+0000 7febca622640 1 -- 172.21.15.7:0/3386136446 >> [v2:172.21.15.83:6818/35294,v1:172.21.15.83:6819/35294] conn(0x7feba80729d0 msgr2=0x7feba8074df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.896+0000 7febca622640 1 --2- 172.21.15.7:0/3386136446 >> [v2:172.21.15.83:6818/35294,v1:172.21.15.83:6819/35294] conn(0x7feba80729d0 0x7feba8074df0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.896+0000 7febca622640 20 client.4326.objecter put_session s=0x7feba8072300 osd=4 1 2024-03-28T16:15:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.896+0000 7febca622640 20 client.4326.objecter shutdown clearing up homeless session... 2024-03-28T16:15:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.896+0000 7febca622640 10 client.4326.objecter successfully canceled tick 2024-03-28T16:15:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.897+0000 7febca622640 1 -- 172.21.15.7:0/3386136446 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7feba806c5f0 msgr2=0x7feba806eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.897+0000 7febca622640 1 --2- 172.21.15.7:0/3386136446 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7feba806c5f0 0x7feba806eab0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7febc0035000 tx=0x7febc0031000 comp rx=0 tx=0).stop 2024-03-28T16:15:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.897+0000 7febca622640 10 monclient: shutdown 2024-03-28T16:15:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.897+0000 7febca622640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.897+0000 7febca622640 1 -- 172.21.15.7:0/3386136446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febc41d5a40 msgr2=0x7febc41d5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.897+0000 7febca622640 1 --2- 172.21.15.7:0/3386136446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febc41d5a40 0x7febc41d5ef0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7febc40ad070 tx=0x7febbc00fc00 comp rx=0 tx=0).stop 2024-03-28T16:15:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.897+0000 7febc9e21640 1 -- 172.21.15.7:0/3386136446 reap_dead start 2024-03-28T16:15:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.899+0000 7febca622640 1 -- 172.21.15.7:0/3386136446 shutdown_connections 2024-03-28T16:15:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.899+0000 7febca622640 1 -- 172.21.15.7:0/3386136446 >> 172.21.15.7:0/3386136446 conn(0x7febc40b2cb0 msgr2=0x7febc40ae360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.899+0000 7febca622640 1 -- 172.21.15.7:0/3386136446 shutdown_connections 2024-03-28T16:15:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.899+0000 7febca622640 1 -- 172.21.15.7:0/3386136446 wait complete. 2024-03-28T16:15:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.899+0000 7febca622640 1 librados: shutdown 2024-03-28T16:15:50.909 INFO:teuthology.orchestra.run.smithi007.stdout:98784247815 2024-03-28T16:15:50.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-28T16:15:50.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.909+0000 7f6c057fa640 1 -- 172.21.15.7:0/312124861 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6bfc053a00 con 0x7f6c0002ec00 2024-03-28T16:15:50.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.909+0000 7f6c057fa640 10 monclient: handle_get_version_reply finishing 1 version 28 2024-03-28T16:15:50.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.909+0000 7f7a9285c640 10 client.4323.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-28T16:15:50.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.909+0000 7f7a9285c640 15 client.4323.objecter _session_command_op_assign -1 2 2024-03-28T16:15:50.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.909+0000 7f7a9285c640 20 client.4323.objecter _get_session s=0x7f7a78072300 osd=6 3 2024-03-28T16:15:50.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.909+0000 7f7a9285c640 20 client.4323.objecter put_session s=0x7f7a78072300 osd=6 3 2024-03-28T16:15:50.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.909+0000 7f7a9285c640 20 client.4323.objecter _get_session s=0x7f7a78072300 osd=6 3 2024-03-28T16:15:50.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.909+0000 7f7a9285c640 15 client.4323.objecter _session_command_op_remove -1 2 2024-03-28T16:15:50.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.909+0000 7f7a9285c640 20 client.4323.objecter get_session s=0x7f7a78072300 osd=6 3 2024-03-28T16:15:50.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.909+0000 7f7a9285c640 15 client.4323.objecter _session_command_op_assign 6 2 2024-03-28T16:15:50.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.909+0000 7f7a9285c640 20 client.4323.objecter put_session s=0x7f7a78072300 osd=6 4 2024-03-28T16:15:50.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.909+0000 7f7a9285c640 10 client.4323.objecter _send_command 2 2024-03-28T16:15:50.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.909+0000 7f7a9285c640 1 -- 172.21.15.7:0/1766437136 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7a8c142990 con 0x7f7a780729d0 2024-03-28T16:15:50.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.912+0000 7f075b037640 10 client.4324.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-28T16:15:50.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.912+0000 7f075b037640 15 client.4324.objecter _session_command_op_assign -1 2 2024-03-28T16:15:50.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.912+0000 7f075b037640 20 client.4324.objecter _get_session s=0x7f07400722f0 osd=1 3 2024-03-28T16:15:50.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.912+0000 7f075b037640 20 client.4324.objecter put_session s=0x7f07400722f0 osd=1 3 2024-03-28T16:15:50.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.912+0000 7f075b037640 20 client.4324.objecter _get_session s=0x7f07400722f0 osd=1 3 2024-03-28T16:15:50.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.912+0000 7f075b037640 15 client.4324.objecter _session_command_op_remove -1 2 2024-03-28T16:15:50.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.912+0000 7f075b037640 20 client.4324.objecter get_session s=0x7f07400722f0 osd=1 3 2024-03-28T16:15:50.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.912+0000 7f075b037640 15 client.4324.objecter _session_command_op_assign 1 2 2024-03-28T16:15:50.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.912+0000 7f075b037640 20 client.4324.objecter put_session s=0x7f07400722f0 osd=1 4 2024-03-28T16:15:50.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.912+0000 7f075b037640 10 client.4324.objecter _send_command 2 2024-03-28T16:15:50.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.912+0000 7f075b037640 1 -- 172.21.15.7:0/2546658315 --> [v2:172.21.15.7:6800/34919,v1:172.21.15.7:6801/34919] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0754140d60 con 0x7f07400729c0 2024-03-28T16:15:50.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.915+0000 7f7a88ff9640 1 -- 172.21.15.7:0/1766437136 <== osd.6 v2:172.21.15.83:6810/35297 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7a8400a720 con 0x7f7a780729d0 2024-03-28T16:15:50.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.915+0000 7f7a88ff9640 10 client.4323.objecter ms_dispatch 0x7f7a8c000b90 command_reply(tid 2: 0 ) v1 2024-03-28T16:15:50.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.915+0000 7f7a88ff9640 10 client.4323.objecter _finish_command 2 = system:0 2024-03-28T16:15:50.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.915+0000 7f7a88ff9640 20 client.4323.objecter put_session s=0x7f7a78072300 osd=6 4 2024-03-28T16:15:50.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.915+0000 7f7a88ff9640 15 client.4323.objecter _session_command_op_remove 6 2 2024-03-28T16:15:50.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.915+0000 7f7a9285c640 10 librados: watch_flush enter 2024-03-28T16:15:50.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.916+0000 7f7a9285c640 10 librados: watch_flush exit 2024-03-28T16:15:50.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.916+0000 7f7a9285c640 10 client.4323.objecter close_session for osd.6 2024-03-28T16:15:50.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.916+0000 7f7a9285c640 1 -- 172.21.15.7:0/1766437136 >> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] conn(0x7f7a780729d0 msgr2=0x7f7a78074df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.916+0000 7f7a9285c640 1 --2- 172.21.15.7:0/1766437136 >> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] conn(0x7f7a780729d0 0x7f7a78074df0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.916+0000 7f7a9285c640 20 client.4323.objecter put_session s=0x7f7a78072300 osd=6 1 2024-03-28T16:15:50.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.916+0000 7f7a9285c640 20 client.4323.objecter shutdown clearing up homeless session... 2024-03-28T16:15:50.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.916+0000 7f7a9285c640 10 client.4323.objecter successfully canceled tick 2024-03-28T16:15:50.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.916+0000 7f7a9285c640 1 -- 172.21.15.7:0/1766437136 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f7a7806c5f0 msgr2=0x7f7a7806eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.916+0000 7f7a9285c640 1 --2- 172.21.15.7:0/1766437136 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f7a7806c5f0 0x7f7a7806eab0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c036000 tx=0x7f7a7c030000 comp rx=0 tx=0).stop 2024-03-28T16:15:50.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.916+0000 7f7a9285c640 10 monclient: shutdown 2024-03-28T16:15:50.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.916+0000 7f7a9285c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:50.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.916+0000 7f7a9285c640 1 -- 172.21.15.7:0/1766437136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a8c0b1890 msgr2=0x7f7a8c144ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.916+0000 7f7a9285c640 1 --2- 172.21.15.7:0/1766437136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a8c0b1890 0x7f7a8c144ba0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000b500 tx=0x7f7a8000b9c0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.917+0000 7f7a90df9640 1 -- 172.21.15.7:0/1766437136 reap_dead start 2024-03-28T16:15:50.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.917+0000 7f7a9285c640 1 -- 172.21.15.7:0/1766437136 shutdown_connections 2024-03-28T16:15:50.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.917+0000 7f7a9285c640 1 -- 172.21.15.7:0/1766437136 >> 172.21.15.7:0/1766437136 conn(0x7f7a8c0b3db0 msgr2=0x7f7a8c13e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:50.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.919+0000 7f7a9285c640 1 -- 172.21.15.7:0/1766437136 shutdown_connections 2024-03-28T16:15:50.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.919+0000 7f7a9285c640 1 -- 172.21.15.7:0/1766437136 wait complete. 2024-03-28T16:15:50.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.920+0000 7f7a9285c640 1 librados: shutdown 2024-03-28T16:15:50.921 INFO:teuthology.orchestra.run.smithi007.stdout:60129542150 2024-03-28T16:15:50.922 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-28T16:15:50.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.922+0000 7f07517fa640 1 -- 172.21.15.7:0/2546658315 <== osd.1 v2:172.21.15.7:6800/34919 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0754140d60 con 0x7f07400729c0 2024-03-28T16:15:50.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.922+0000 7f07517fa640 10 client.4324.objecter ms_dispatch 0x7f0754000b90 command_reply(tid 2: 0 ) v1 2024-03-28T16:15:50.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.922+0000 7f07517fa640 10 client.4324.objecter _finish_command 2 = system:0 2024-03-28T16:15:50.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.922+0000 7f07517fa640 20 client.4324.objecter put_session s=0x7f07400722f0 osd=1 4 2024-03-28T16:15:50.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.922+0000 7f07517fa640 15 client.4324.objecter _session_command_op_remove 1 2 2024-03-28T16:15:50.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.922+0000 7f075b037640 10 librados: watch_flush enter 2024-03-28T16:15:50.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.923+0000 7f075b037640 10 librados: watch_flush exit 2024-03-28T16:15:50.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.923+0000 7f075b037640 10 client.4324.objecter close_session for osd.1 2024-03-28T16:15:50.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.923+0000 7f075b037640 1 -- 172.21.15.7:0/2546658315 >> [v2:172.21.15.7:6800/34919,v1:172.21.15.7:6801/34919] conn(0x7f07400729c0 msgr2=0x7f0740074de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.923+0000 7f075b037640 1 --2- 172.21.15.7:0/2546658315 >> [v2:172.21.15.7:6800/34919,v1:172.21.15.7:6801/34919] conn(0x7f07400729c0 0x7f0740074de0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.923+0000 7f075b037640 20 client.4324.objecter put_session s=0x7f07400722f0 osd=1 1 2024-03-28T16:15:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.923+0000 7f075b037640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-28T16:15:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.923+0000 7f075b037640 10 client.4324.objecter successfully canceled tick 2024-03-28T16:15:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.923+0000 7f075b037640 1 -- 172.21.15.7:0/2546658315 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f074006c5e0 msgr2=0x7f074006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.923+0000 7f075b037640 1 --2- 172.21.15.7:0/2546658315 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f074006c5e0 0x7f074006eaa0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0748001610 tx=0x7f0748007930 comp rx=0 tx=0).stop 2024-03-28T16:15:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.923+0000 7f075b037640 10 monclient: shutdown 2024-03-28T16:15:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.923+0000 7f075b037640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.923+0000 7f075b037640 1 -- 172.21.15.7:0/2546658315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075413e000 msgr2=0x7f07541353a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.923+0000 7f075b037640 1 --2- 172.21.15.7:0/2546658315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f075413e000 0x7f07541353a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f074c00c7b0 tx=0x7f074c00cc70 comp rx=0 tx=0).stop 2024-03-28T16:15:50.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.923+0000 7f07595d4640 1 -- 172.21.15.7:0/2546658315 reap_dead start 2024-03-28T16:15:50.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.923+0000 7f075b037640 1 -- 172.21.15.7:0/2546658315 shutdown_connections 2024-03-28T16:15:50.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.923+0000 7f075b037640 1 -- 172.21.15.7:0/2546658315 >> 172.21.15.7:0/2546658315 conn(0x7f07540b2ad0 msgr2=0x7f07541c6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.923+0000 7f075b037640 1 -- 172.21.15.7:0/2546658315 shutdown_connections 2024-03-28T16:15:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.925+0000 7f075b037640 1 -- 172.21.15.7:0/2546658315 wait complete. 2024-03-28T16:15:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.924+0000 7f6c057fa640 1 -- 172.21.15.7:0/312124861 <== osd.2 v2:172.21.15.7:6824/34920 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6bec075470 con 0x7f6bec0729a0 2024-03-28T16:15:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.924+0000 7f6c057fa640 10 client.4333.objecter ms_dispatch 0x7f6c08000b90 command_reply(tid 1: 0 ) v1 2024-03-28T16:15:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.924+0000 7f6c057fa640 10 client.4333.objecter _finish_command 1 = system:0 2024-03-28T16:15:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.924+0000 7f6c057fa640 20 client.4333.objecter put_session s=0x7f6bec0722d0 osd=2 4 2024-03-28T16:15:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.924+0000 7f6c057fa640 15 client.4333.objecter _session_command_op_remove 2 1 2024-03-28T16:15:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.925+0000 7f075b037640 1 librados: shutdown 2024-03-28T16:15:50.941 INFO:teuthology.orchestra.run.smithi007.stdout:90194313227 2024-03-28T16:15:50.941 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-28T16:15:50.944 INFO:teuthology.orchestra.run.smithi007.stdout:73014444038 2024-03-28T16:15:50.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-28T16:15:50.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.952+0000 7f6c0ed37640 10 client.4333.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-28T16:15:50.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.952+0000 7f6c0ed37640 15 client.4333.objecter _session_command_op_assign -1 2 2024-03-28T16:15:50.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.952+0000 7f6c0ed37640 20 client.4333.objecter _get_session s=0x7f6bec0722d0 osd=2 3 2024-03-28T16:15:50.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.952+0000 7f6c0ed37640 20 client.4333.objecter put_session s=0x7f6bec0722d0 osd=2 3 2024-03-28T16:15:50.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.952+0000 7f6c0ed37640 20 client.4333.objecter _get_session s=0x7f6bec0722d0 osd=2 3 2024-03-28T16:15:50.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.952+0000 7f6c0ed37640 15 client.4333.objecter _session_command_op_remove -1 2 2024-03-28T16:15:50.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.952+0000 7f6c0ed37640 20 client.4333.objecter get_session s=0x7f6bec0722d0 osd=2 3 2024-03-28T16:15:50.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.952+0000 7f6c0ed37640 15 client.4333.objecter _session_command_op_assign 2 2 2024-03-28T16:15:50.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.952+0000 7f6c0ed37640 20 client.4333.objecter put_session s=0x7f6bec0722d0 osd=2 4 2024-03-28T16:15:50.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.952+0000 7f6c0ed37640 10 client.4333.objecter _send_command 2 2024-03-28T16:15:50.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.952+0000 7f6c0ed37640 1 -- 172.21.15.7:0/312124861 --> [v2:172.21.15.7:6824/34920,v1:172.21.15.7:6825/34920] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6c000cf190 con 0x7f6bec0729a0 2024-03-28T16:15:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.967+0000 7f6c057fa640 1 -- 172.21.15.7:0/312124861 <== osd.2 v2:172.21.15.7:6824/34920 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6c000cf190 con 0x7f6bec0729a0 2024-03-28T16:15:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.968+0000 7f6c057fa640 10 client.4333.objecter ms_dispatch 0x7f6c08000b90 command_reply(tid 2: 0 ) v1 2024-03-28T16:15:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.968+0000 7f6c057fa640 10 client.4333.objecter _finish_command 2 = system:0 2024-03-28T16:15:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.968+0000 7f6c057fa640 20 client.4333.objecter put_session s=0x7f6bec0722d0 osd=2 4 2024-03-28T16:15:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.968+0000 7f6c057fa640 15 client.4333.objecter _session_command_op_remove 2 2 2024-03-28T16:15:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.968+0000 7f6c0ed37640 10 librados: watch_flush enter 2024-03-28T16:15:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.968+0000 7f6c0ed37640 10 librados: watch_flush exit 2024-03-28T16:15:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.968+0000 7f6c0ed37640 10 client.4333.objecter close_session for osd.2 2024-03-28T16:15:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.968+0000 7f6c0ed37640 1 -- 172.21.15.7:0/312124861 >> [v2:172.21.15.7:6824/34920,v1:172.21.15.7:6825/34920] conn(0x7f6bec0729a0 msgr2=0x7f6bec074dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.968+0000 7f6c0ed37640 1 --2- 172.21.15.7:0/312124861 >> [v2:172.21.15.7:6824/34920,v1:172.21.15.7:6825/34920] conn(0x7f6bec0729a0 0x7f6bec074dc0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.968+0000 7f6c0ed37640 20 client.4333.objecter put_session s=0x7f6bec0722d0 osd=2 1 2024-03-28T16:15:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.968+0000 7f6c0ed37640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-28T16:15:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.968+0000 7f6c0ed37640 10 client.4333.objecter successfully canceled tick 2024-03-28T16:15:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.969+0000 7f6c0ed37640 1 -- 172.21.15.7:0/312124861 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f6bec06c5c0 msgr2=0x7f6bec06ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.969+0000 7f6c0ed37640 1 --2- 172.21.15.7:0/312124861 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f6bec06c5c0 0x7f6bec06ea80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6c08000b60 tx=0x7f6c0806c040 comp rx=0 tx=0).stop 2024-03-28T16:15:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.969+0000 7f6c0ed37640 10 monclient: shutdown 2024-03-28T16:15:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.969+0000 7f6c0ed37640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.969+0000 7f6c0ed37640 1 -- 172.21.15.7:0/312124861 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c0002ec00 msgr2=0x7f6c00029c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.969+0000 7f6c0ed37640 1 --2- 172.21.15.7:0/312124861 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c0002ec00 0x7f6c00029c50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc00c410 tx=0x7f6bfc00c8d0 comp rx=0 tx=0).stop 2024-03-28T16:15:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.969+0000 7f6c0d2d4640 1 -- 172.21.15.7:0/312124861 reap_dead start 2024-03-28T16:15:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.969+0000 7f6c0ed37640 1 -- 172.21.15.7:0/312124861 shutdown_connections 2024-03-28T16:15:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.969+0000 7f6c0ed37640 1 -- 172.21.15.7:0/312124861 >> 172.21.15.7:0/312124861 conn(0x7f6c00036600 msgr2=0x7f6c00031e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.969+0000 7f6c0ed37640 1 -- 172.21.15.7:0/312124861 shutdown_connections 2024-03-28T16:15:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.969+0000 7f6c0ed37640 1 -- 172.21.15.7:0/312124861 wait complete. 2024-03-28T16:15:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:50.969+0000 7f6c0ed37640 1 librados: shutdown 2024-03-28T16:15:50.990 INFO:teuthology.orchestra.run.smithi007.stdout:73014444040 2024-03-28T16:15:50.991 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-28T16:15:51.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.061+0000 7fe79d020640 10 monclient: get_monmap_and_config 2024-03-28T16:15:51.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.061+0000 7fe79d020640 10 monclient: build_initial_monmap 2024-03-28T16:15:51.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.061+0000 7fe79d020640 10 monclient: monmap: 2024-03-28T16:15:51.063 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:51.063 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.063 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:51.063194+0000 2024-03-28T16:15:51.063 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:51.063194+0000 2024-03-28T16:15:51.063 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:51.064 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.064 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:51.064 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:51.064 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:51.064 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:51.064 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:51.064 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.062+0000 7f9d680af640 10 monclient: get_monmap_and_config 2024-03-28T16:15:51.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.062+0000 7f9d680af640 10 monclient: build_initial_monmap 2024-03-28T16:15:51.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.062+0000 7f9d680af640 10 monclient: monmap: 2024-03-28T16:15:51.065 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:51.065 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.065 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:51.064243+0000 2024-03-28T16:15:51.065 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:51.064243+0000 2024-03-28T16:15:51.065 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:51.065 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.065 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:51.065 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:51.065 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:51.065 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:51.065 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:51.065 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.062+0000 7f16a84db640 10 monclient: get_monmap_and_config 2024-03-28T16:15:51.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.062+0000 7f16a84db640 10 monclient: build_initial_monmap 2024-03-28T16:15:51.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.062+0000 7f16a84db640 10 monclient: monmap: 2024-03-28T16:15:51.066 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:51.066 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.066 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:51.064560+0000 2024-03-28T16:15:51.066 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:51.064560+0000 2024-03-28T16:15:51.066 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:51.066 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.066 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:51.067 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:51.067 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:51.067 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:51.067 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:51.067 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.063+0000 7f16a84db640 1 Processor -- start 2024-03-28T16:15:51.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.063+0000 7f16a84db640 1 -- start start 2024-03-28T16:15:51.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.063+0000 7f16a84db640 10 monclient: init 2024-03-28T16:15:51.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.065+0000 7f16a84db640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:51.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.065+0000 7f16a84db640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-03-28T16:15:51.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.065+0000 7f16a84db640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f16a0071480 0x7f16a013e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.065+0000 7f16a84db640 10 monclient(hunting): picked mon.noname-c con 0x7f16a0071480 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:51.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.065+0000 7f16a84db640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f16a00b32e0 0x7f16a0144a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.065+0000 7f16a84db640 10 monclient(hunting): picked mon.noname-a con 0x7f16a00b32e0 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:51.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.065+0000 7f16a84db640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16a013ee50 0x7f16a0144f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.065+0000 7f16a84db640 10 monclient(hunting): picked mon.noname-b con 0x7f16a013ee50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:51.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.065+0000 7f16a84db640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.065+0000 7f16a84db640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16a0071860 con 0x7f16a00b32e0 2024-03-28T16:15:51.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.065+0000 7f16a84db640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.065+0000 7f16a84db640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16a00a8c90 con 0x7f16a0071480 2024-03-28T16:15:51.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.065+0000 7f16a84db640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.065+0000 7f16a84db640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a0072140 con 0x7f16a013ee50 2024-03-28T16:15:51.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.065+0000 7f16a84db640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:51.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.065+0000 7f16a84db640 10 monclient(hunting): authenticate will time out at 1418.206933s 2024-03-28T16:15:51.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.066+0000 7f16a6277640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f16a0071480 0x7f16a013e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.066+0000 7f16a6a78640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16a013ee50 0x7f16a0144f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.063+0000 7f9d680af640 1 Processor -- start 2024-03-28T16:15:51.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.063+0000 7f9d680af640 1 -- start start 2024-03-28T16:15:51.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.063+0000 7f9d680af640 10 monclient: init 2024-03-28T16:15:51.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.064+0000 7f9d680af640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:51.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.064+0000 7f9d680af640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-03-28T16:15:51.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.064+0000 7f9d680af640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9d60071480 0x7f9d6013e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.064+0000 7f9d680af640 10 monclient(hunting): picked mon.noname-a con 0x7f9d60071480 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:51.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.064+0000 7f9d680af640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9d600b32e0 0x7f9d60144a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.064+0000 7f9d680af640 10 monclient(hunting): picked mon.noname-c con 0x7f9d600b32e0 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:51.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.064+0000 7f9d680af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d6013ee50 addr v1:172.21.15.83:6789/0 2024-03-28T16:15:51.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.064+0000 7f9d680af640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d600a8c90 con 0x7f9d6013ee50 2024-03-28T16:15:51.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.064+0000 7f9d680af640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.064+0000 7f9d680af640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d60071860 con 0x7f9d60071480 2024-03-28T16:15:51.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.064+0000 7f9d680af640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.064+0000 7f9d680af640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d60145650 con 0x7f9d600b32e0 2024-03-28T16:15:51.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.064+0000 7f9d680af640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:51.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.064+0000 7f9d680af640 10 monclient(hunting): authenticate will time out at 1418.206567s 2024-03-28T16:15:51.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.067+0000 7f9d6564a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9d600b32e0 0x7f9d60144a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.067+0000 7f9d6564a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9d600b32e0 0x7f9d60144a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:58092/0 (socket says 172.21.15.7:58092) 2024-03-28T16:15:51.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.067+0000 7f9d6564a640 1 -- 172.21.15.7:0/764467886 learned_addr learned my addr 172.21.15.7:0/764467886 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:51.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.067+0000 7f9d6564a640 10 monclient(hunting): get_auth_request con 0x7f9d600b32e0 auth_method 0 2024-03-28T16:15:51.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.067+0000 7f9d6564a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:51.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.067+0000 7f9d6564a640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:51.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.067+0000 7f9d6564a640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:51.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.067+0000 7f16a6277640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f16a0071480 0x7f16a013e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:58096/0 (socket says 172.21.15.7:58096) 2024-03-28T16:15:51.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.067+0000 7f16a6277640 1 -- 172.21.15.7:0/3674227780 learned_addr learned my addr 172.21.15.7:0/3674227780 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:51.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.067+0000 7f16a6277640 10 monclient(hunting): get_auth_request con 0x7f16a0071480 auth_method 0 2024-03-28T16:15:51.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.067+0000 7f16a6277640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:51.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.067+0000 7f16a6277640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:51.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.067+0000 7f16a6277640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:51.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.067+0000 7f16a6a78640 10 monclient(hunting): get_auth_request con 0x7f16a013ee50 auth_method 0 2024-03-28T16:15:51.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.067+0000 7f16a6a78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:51.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.067+0000 7f16a6a78640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:51.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.067+0000 7f16a6a78640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:51.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.068+0000 7f16a6a78640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:51.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.068+0000 7f16a6a78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:51.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.068+0000 7f16a6a78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:51.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.067+0000 7fe79d020640 1 Processor -- start 2024-03-28T16:15:51.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.067+0000 7fe79d020640 1 -- start start 2024-03-28T16:15:51.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.067+0000 7fe79d020640 10 monclient: init 2024-03-28T16:15:51.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.068+0000 7fe79d020640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:51.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.068+0000 7fe79d020640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-03-28T16:15:51.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.068+0000 7fe79d020640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe79002e1d0 0x7fe79002c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.068+0000 7fe79d020640 10 monclient(hunting): picked mon.noname-a con 0x7fe79002e1d0 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:51.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.068+0000 7fe79d020640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe79002ec10 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:51.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.068+0000 7fe79d020640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe790032ec0 0x7fe7900cc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.068+0000 7fe79d020640 10 monclient(hunting): picked mon.noname-c con 0x7fe790032ec0 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:51.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.068+0000 7fe79d020640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79002fbb0 con 0x7fe79002ec10 2024-03-28T16:15:51.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.068+0000 7fe79d020640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.068+0000 7fe79d020640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe79002d6f0 con 0x7fe79002e1d0 2024-03-28T16:15:51.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.068+0000 7fe79d020640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.068+0000 7fe79d020640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe79002d830 con 0x7fe790032ec0 2024-03-28T16:15:51.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.069+0000 7fe79d020640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:51.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.069+0000 7fe79d020640 10 monclient(hunting): authenticate will time out at 1418.210857s 2024-03-28T16:15:51.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.069+0000 7fe79c81f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe790032ec0 0x7fe7900cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.069+0000 7f9d6564a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:51.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.069+0000 7f9d6564a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:51.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.069+0000 7f9d6564a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:51.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.069+0000 7fe79c81f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe790032ec0 0x7fe7900cc070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:58110/0 (socket says 172.21.15.7:58110) 2024-03-28T16:15:51.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.069+0000 7fe79c81f640 1 -- 172.21.15.7:0/1182495510 learned_addr learned my addr 172.21.15.7:0/1182495510 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:51.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.069+0000 7fe79c81f640 10 monclient(hunting): get_auth_request con 0x7fe790032ec0 auth_method 0 2024-03-28T16:15:51.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.069+0000 7fe79c81f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:51.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.069+0000 7fe79c81f640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:51.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.069+0000 7fe79c81f640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:51.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.070+0000 7f16a6277640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:51.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.070+0000 7f16a6277640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:51.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.070+0000 7f16a6277640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:51.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.071+0000 7f95b99df640 10 monclient: get_monmap_and_config 2024-03-28T16:15:51.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.071+0000 7f95b99df640 10 monclient: build_initial_monmap 2024-03-28T16:15:51.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.071+0000 7f95b99df640 10 monclient: monmap: 2024-03-28T16:15:51.079 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:51.079 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.079 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:51.072799+0000 2024-03-28T16:15:51.079 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:51.072799+0000 2024-03-28T16:15:51.079 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:51.079 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.079 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:51.079 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:51.079 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:51.080 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:51.080 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:51.080 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.072+0000 7fe79c81f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:51.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.072+0000 7fe79c81f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:51.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.072+0000 7fe79c81f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:51.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.072+0000 7f9d6564a640 10 monclient(hunting): handle_auth_done global_id 4336 payload 274 2024-03-28T16:15:51.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.072+0000 7f9d6564a640 1 -- 172.21.15.7:0/764467886 >> v2:172.21.15.7:3300/0 conn(0x7f9d60071480 msgr2=0x7f9d6013e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:51.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.072+0000 7f9d6564a640 1 --2- 172.21.15.7:0/764467886 >> v2:172.21.15.7:3300/0 conn(0x7f9d60071480 0x7f9d6013e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.072+0000 7f9d6564a640 1 -- 172.21.15.7:0/764467886 >> v1:172.21.15.83:6789/0 conn(0x7f9d6013ee50 legacy=0x7f9d60144f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.072+0000 7f9d6564a640 10 monclient: _finish_hunting 0 2024-03-28T16:15:51.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.072+0000 7f9d6564a640 1 monclient: found mon.noname-c 2024-03-28T16:15:51.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.072+0000 7f9d6564a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:51.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.072+0000 7f95b99df640 1 Processor -- start 2024-03-28T16:15:51.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.072+0000 7f95b99df640 1 -- start start 2024-03-28T16:15:51.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.072+0000 7f95b99df640 10 monclient: init 2024-03-28T16:15:51.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.073+0000 7f95b99df640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:51.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.073+0000 7f95b99df640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-03-28T16:15:51.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.073+0000 7f95b99df640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f95ac038fe0 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:51.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.073+0000 7f95b99df640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f95ac035640 0x7f95ac0cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.073+0000 7f95b99df640 10 monclient(hunting): picked mon.noname-c con 0x7f95ac035640 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:51.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.073+0000 7f95b99df640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f95ac0cc600 addr v1:172.21.15.83:6789/0 2024-03-28T16:15:51.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.073+0000 7f95b99df640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ac035bf0 con 0x7f95ac038fe0 2024-03-28T16:15:51.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.073+0000 7f95b99df640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ac0373e0 con 0x7f95ac0cc600 2024-03-28T16:15:51.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.073+0000 7f95b99df640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.073+0000 7f95b99df640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95ac0c9d50 con 0x7f95ac035640 2024-03-28T16:15:51.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.073+0000 7f95b99df640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:51.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.073+0000 7f95b99df640 10 monclient(hunting): authenticate will time out at 1418.215378s 2024-03-28T16:15:51.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.072+0000 7f9d6564a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3300/0 2024-03-28T16:15:51.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.072+0000 7f9d6564a640 1 -- 172.21.15.7:0/764467886 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d60072140 con 0x7f9d600b32e0 2024-03-28T16:15:51.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.074+0000 7f16a6a78640 10 monclient(hunting): handle_auth_done global_id 4283 payload 274 2024-03-28T16:15:51.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.074+0000 7f16a6a78640 1 -- 172.21.15.7:0/3674227780 >> v2:172.21.15.83:3300/0 conn(0x7f16a0071480 msgr2=0x7f16a013e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:51.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.074+0000 7f16a6a78640 1 --2- 172.21.15.7:0/3674227780 >> v2:172.21.15.83:3300/0 conn(0x7f16a0071480 0x7f16a013e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.074+0000 7f16a6a78640 1 -- 172.21.15.7:0/3674227780 >> v2:172.21.15.7:3300/0 conn(0x7f16a00b32e0 msgr2=0x7f16a0144a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:51.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.074+0000 7f16a6a78640 1 --2- 172.21.15.7:0/3674227780 >> v2:172.21.15.7:3300/0 conn(0x7f16a00b32e0 0x7f16a0144a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.074+0000 7f16a6a78640 10 monclient: _finish_hunting 0 2024-03-28T16:15:51.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.074+0000 7f16a6a78640 1 monclient: found mon.noname-b 2024-03-28T16:15:51.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.074+0000 7f16a6a78640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:51.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.074+0000 7f16a6a78640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-03-28T16:15:51.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.074+0000 7f16a6a78640 1 -- 172.21.15.7:0/3674227780 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a00722b0 con 0x7f16a013ee50 2024-03-28T16:15:51.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.074+0000 7fe79c81f640 10 monclient(hunting): handle_auth_done global_id 4342 payload 274 2024-03-28T16:15:51.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.074+0000 7fe79c81f640 1 -- 172.21.15.7:0/1182495510 >> v2:172.21.15.7:3300/0 conn(0x7fe79002e1d0 msgr2=0x7fe79002c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:51.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.074+0000 7fe79c81f640 1 --2- 172.21.15.7:0/1182495510 >> v2:172.21.15.7:3300/0 conn(0x7fe79002e1d0 0x7fe79002c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.074+0000 7fe79c81f640 1 -- 172.21.15.7:0/1182495510 >> v1:172.21.15.7:6789/0 conn(0x7fe79002ec10 legacy=0x7fe7900327b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:51.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.074+0000 7fe79c81f640 10 monclient: _finish_hunting 0 2024-03-28T16:15:51.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.074+0000 7fe79c81f640 1 monclient: found mon.noname-c 2024-03-28T16:15:51.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.074+0000 7fe79c81f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:51.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.074+0000 7fe79c81f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3300/0 2024-03-28T16:15:51.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.074+0000 7fe79c81f640 1 -- 172.21.15.7:0/1182495510 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79002e5b0 con 0x7fe790032ec0 2024-03-28T16:15:51.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.077+0000 7f16a6a78640 1 --2- 172.21.15.7:0/3674227780 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16a013ee50 0x7f16a0144f40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f16a00b4a10 tx=0x7f169802bd00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bc780d75e9a4569 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.079+0000 7f9d6564a640 1 --2- 172.21.15.7:0/764467886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d600b32e0 0x7f9d60144a00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9d60071ac0 tx=0x7f9d5c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c47e594b74563cc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.080+0000 7f95b3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f95ac035640 0x7f95ac0cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.081+0000 7f95b3fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f95ac035640 0x7f95ac0cc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:58114/0 (socket says 172.21.15.7:58114) 2024-03-28T16:15:51.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.081+0000 7f95b3fff640 1 -- 172.21.15.7:0/4240112885 learned_addr learned my addr 172.21.15.7:0/4240112885 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:51.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.081+0000 7f95b3fff640 10 monclient(hunting): get_auth_request con 0x7f95ac035640 auth_method 0 2024-03-28T16:15:51.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.081+0000 7f95b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:51.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.081+0000 7f95b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:51.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.081+0000 7f95b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:51.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.081+0000 7fe79c81f640 1 --2- 172.21.15.7:0/1182495510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe790032ec0 0x7fe7900cc070 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe7900a5010 tx=0x7fe78402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4aa10070f5c546d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.082+0000 7f16a5275640 1 -- 172.21.15.7:0/3674227780 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169802cab0 con 0x7f16a013ee50 2024-03-28T16:15:51.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.082+0000 7f16a5275640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.082+0000 7f16a5275640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:51.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.082+0000 7f16a5275640 10 monclient: dump: 2024-03-28T16:15:51.092 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.092 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.092 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.092 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.092 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.092 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.092 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.093 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.093 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.093 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.082+0000 7f16a5275640 10 monclient: _finish_auth 0 2024-03-28T16:15:51.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.082+0000 7f16a5275640 10 monclient: _check_auth_tickets 2024-03-28T16:15:51.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.082+0000 7f16a5275640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:51.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.082+0000 7f16a5275640 1 -- 172.21.15.7:0/3674227780 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169802cc10 con 0x7f16a013ee50 2024-03-28T16:15:51.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.082+0000 7f16a5275640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:51.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.082+0000 7f16a5275640 1 -- 172.21.15.7:0/3674227780 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1698034440 con 0x7f16a013ee50 2024-03-28T16:15:51.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.082+0000 7f16a5275640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.082+0000 7f16a5275640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:15:51.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.082+0000 7f16a5275640 10 monclient: dump: 2024-03-28T16:15:51.093 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.093 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.094 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.094 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.094 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.094 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.094 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.094 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.094 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.094 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.083+0000 7f16a84db640 5 monclient: authenticate success, global_id 4283 2024-03-28T16:15:51.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.083+0000 7f16a84db640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:51.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.083+0000 7f16a84db640 4 set_mon_vals no callback set 2024-03-28T16:15:51.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.083+0000 7f16a84db640 10 monclient: shutdown 2024-03-28T16:15:51.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.083+0000 7f16a84db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:51.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.083+0000 7f16a84db640 1 -- 172.21.15.7:0/3674227780 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16a013ee50 msgr2=0x7f16a0144f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.083+0000 7f16a84db640 1 --2- 172.21.15.7:0/3674227780 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16a013ee50 0x7f16a0144f40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f16a00b4a10 tx=0x7f169802bd00 comp rx=0 tx=0).stop 2024-03-28T16:15:51.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.084+0000 7f16a84db640 1 -- 172.21.15.7:0/3674227780 shutdown_connections 2024-03-28T16:15:51.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.084+0000 7f16a84db640 1 --2- 172.21.15.7:0/3674227780 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16a013ee50 0x7f16a0144f40 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.084+0000 7f16a84db640 1 --2- 172.21.15.7:0/3674227780 >> v2:172.21.15.7:3300/0 conn(0x7f16a00b32e0 0x7f16a0144a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.084+0000 7f16a84db640 1 --2- 172.21.15.7:0/3674227780 >> v2:172.21.15.83:3300/0 conn(0x7f16a0071480 0x7f16a013e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.084+0000 7f16a84db640 1 -- 172.21.15.7:0/3674227780 >> 172.21.15.7:0/3674227780 conn(0x7f16a00b70e0 msgr2=0x7f16a00b74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:51.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.084+0000 7f16a84db640 1 -- 172.21.15.7:0/3674227780 shutdown_connections 2024-03-28T16:15:51.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.084+0000 7f16a84db640 1 -- 172.21.15.7:0/3674227780 wait complete. 2024-03-28T16:15:51.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.085+0000 7f95b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:51.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.085+0000 7f95b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:51.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.085+0000 7f95b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:51.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.085+0000 7f16a84db640 10 monclient: build_initial_monmap 2024-03-28T16:15:51.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.085+0000 7f16a84db640 10 monclient: monmap: 2024-03-28T16:15:51.098 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:51.098 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:51.098 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:51.098 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:51.098 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:51.098 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.098 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:51.098 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:51.098 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:51.098 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.085+0000 7f16a84db640 1 librados: starting msgr at 2024-03-28T16:15:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.085+0000 7f16a84db640 1 librados: starting objecter 2024-03-28T16:15:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.086+0000 7f16a84db640 1 Processor -- start 2024-03-28T16:15:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.086+0000 7f16a84db640 1 -- start start 2024-03-28T16:15:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.086+0000 7f16a84db640 1 librados: setting wanted keys 2024-03-28T16:15:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.086+0000 7f16a84db640 1 librados: calling monclient init 2024-03-28T16:15:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.086+0000 7f16a84db640 10 monclient: init 2024-03-28T16:15:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.086+0000 7f16a84db640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.086+0000 7f16a84db640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-28T16:15:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.087+0000 7f16a84db640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16a0071480 0x7f16a013fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.087+0000 7f16a84db640 10 monclient(hunting): picked mon.noname-a con 0x7f16a0071480 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.087+0000 7f16a84db640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16a00b32e0 0x7f16a00afd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.087+0000 7f16a84db640 10 monclient(hunting): picked mon.noname-c con 0x7f16a00b32e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.087+0000 7f16a84db640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16a013ee50 0x7f16a00b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.087+0000 7f16a84db640 10 monclient(hunting): picked mon.noname-b con 0x7f16a013ee50 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.087+0000 7f16a84db640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.087+0000 7f16a84db640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a0140ed0 con 0x7f16a0071480 2024-03-28T16:15:51.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.087+0000 7f16a84db640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.087+0000 7f16a84db640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a00b07f0 con 0x7f16a013ee50 2024-03-28T16:15:51.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.087+0000 7f16a84db640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.087+0000 7f16a84db640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a013f2c0 con 0x7f16a00b32e0 2024-03-28T16:15:51.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.087+0000 7f16a84db640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:51.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.087+0000 7f16a84db640 10 monclient(hunting): authenticate will time out at 1418.229262s 2024-03-28T16:15:51.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.087+0000 7f16a5a76640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16a00b32e0 0x7f16a00afd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.088+0000 7f16a5a76640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16a00b32e0 0x7f16a00afd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60228/0 (socket says 172.21.15.7:60228) 2024-03-28T16:15:51.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.088+0000 7f16a5a76640 1 -- 172.21.15.7:0/2557141569 learned_addr learned my addr 172.21.15.7:0/2557141569 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:51.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.088+0000 7f16a5a76640 10 monclient(hunting): get_auth_request con 0x7f16a00b32e0 auth_method 0 2024-03-28T16:15:51.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.088+0000 7f16a5a76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:51.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.088+0000 7f16a5a76640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:51.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.088+0000 7f16a5a76640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:51.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.088+0000 7f95b3fff640 10 monclient(hunting): handle_auth_done global_id 4348 payload 274 2024-03-28T16:15:51.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.088+0000 7f95b3fff640 1 -- 172.21.15.7:0/4240112885 >> v1:172.21.15.83:6789/0 conn(0x7f95ac0cc600 legacy=0x7f95ac0d6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.088+0000 7f95b3fff640 1 -- 172.21.15.7:0/4240112885 >> v1:172.21.15.7:6789/0 conn(0x7f95ac038fe0 legacy=0x7f95ac0c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:51.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.088+0000 7f95b3fff640 10 monclient: _finish_hunting 0 2024-03-28T16:15:51.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.088+0000 7f95b3fff640 1 monclient: found mon.noname-c 2024-03-28T16:15:51.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.088+0000 7f95b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:51.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.088+0000 7f95b3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3300/0 2024-03-28T16:15:51.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.088+0000 7f95b3fff640 1 -- 172.21.15.7:0/4240112885 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ac037650 con 0x7f95ac035640 2024-03-28T16:15:51.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.089+0000 7f3fd5f37640 10 monclient: get_monmap_and_config 2024-03-28T16:15:51.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.089+0000 7f3fd5f37640 10 monclient: build_initial_monmap 2024-03-28T16:15:51.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.089+0000 7f3fd5f37640 10 monclient: monmap: 2024-03-28T16:15:51.102 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:51.102 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.102 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:51.090872+0000 2024-03-28T16:15:51.102 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:51.090872+0000 2024-03-28T16:15:51.102 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:51.102 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.102 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:51.102 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:51.103 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:51.103 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:51.103 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:51.103 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.089+0000 7f16a5a76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:51.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.089+0000 7f16a5a76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:51.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.089+0000 7f16a5a76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:51.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.090+0000 7f3fd5f37640 1 Processor -- start 2024-03-28T16:15:51.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.090+0000 7f3fd5f37640 1 -- start start 2024-03-28T16:15:51.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.090+0000 7f3fd5f37640 10 monclient: init 2024-03-28T16:15:51.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.090+0000 7fe796ffd640 1 -- 172.21.15.7:0/1182495510 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7840317e0 con 0x7fe790032ec0 2024-03-28T16:15:51.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.090+0000 7fe796ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.090+0000 7fe796ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:51.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7fe796ffd640 10 monclient: dump: 2024-03-28T16:15:51.105 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.105 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.105 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.105 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.105 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.105 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.105 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.106 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.106 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.106 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7fe796ffd640 10 monclient: _finish_auth 0 2024-03-28T16:15:51.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7fe796ffd640 10 monclient: _check_auth_tickets 2024-03-28T16:15:51.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7fe796ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:51.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7fe79d020640 5 monclient: authenticate success, global_id 4342 2024-03-28T16:15:51.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7fe79d020640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:51.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7fe796ffd640 1 -- 172.21.15.7:0/1182495510 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe784031940 con 0x7fe790032ec0 2024-03-28T16:15:51.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7fe796ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:51.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7f3fd5f37640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:51.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7f3fd5f37640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-03-28T16:15:51.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7f3fd5f37640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3fd0071480 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:51.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7f3fd5f37640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3fd00b32e0 0x7f3fd0144a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7f3fd5f37640 10 monclient(hunting): picked mon.noname-c con 0x7f3fd00b32e0 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:51.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7f3fd5f37640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3fd00a86e0 0x7f3fd0144f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7f3fd5f37640 10 monclient(hunting): picked mon.noname-b con 0x7f3fd00a86e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:51.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7f3fd5f37640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd00a8c90 con 0x7f3fd0071480 2024-03-28T16:15:51.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7f3fd5f37640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7f3fd5f37640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fd0071860 con 0x7f3fd00b32e0 2024-03-28T16:15:51.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7f3fd5f37640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7f3fd5f37640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd013f020 con 0x7f3fd00a86e0 2024-03-28T16:15:51.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7f3fd5f37640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:51.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7f3fd5f37640 10 monclient(hunting): authenticate will time out at 1418.233731s 2024-03-28T16:15:51.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7fe79d020640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:51.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7fe79d020640 4 set_mon_vals no callback set 2024-03-28T16:15:51.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7fe796ffd640 1 -- 172.21.15.7:0/1182495510 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe784031cd0 con 0x7fe790032ec0 2024-03-28T16:15:51.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7fe796ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7fe796ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:51.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7fe796ffd640 10 monclient: dump: 2024-03-28T16:15:51.109 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.109 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.109 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.110 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.110 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.110 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.110 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.110 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.110 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.110 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.091+0000 7f16a6a78640 1 --2- 172.21.15.7:0/2557141569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16a013ee50 0x7f16a00b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.092+0000 7fe79d020640 10 monclient: shutdown 2024-03-28T16:15:51.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.092+0000 7fe79d020640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:51.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.092+0000 7fe79d020640 1 -- 172.21.15.7:0/1182495510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe790032ec0 msgr2=0x7fe7900cc070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.092+0000 7fe79d020640 1 --2- 172.21.15.7:0/1182495510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe790032ec0 0x7fe7900cc070 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe7900a5010 tx=0x7fe78402b880 comp rx=0 tx=0).stop 2024-03-28T16:15:51.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.092+0000 7f16a6a78640 10 monclient(hunting): get_auth_request con 0x7f16a013ee50 auth_method 0 2024-03-28T16:15:51.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.092+0000 7fe79d020640 1 -- 172.21.15.7:0/1182495510 shutdown_connections 2024-03-28T16:15:51.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.092+0000 7fe79d020640 1 --2- 172.21.15.7:0/1182495510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe790032ec0 0x7fe7900cc070 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.092+0000 7fe79d020640 1 --2- 172.21.15.7:0/1182495510 >> v2:172.21.15.7:3300/0 conn(0x7fe79002e1d0 0x7fe79002c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.092+0000 7fe79d020640 1 -- 172.21.15.7:0/1182495510 >> 172.21.15.7:0/1182495510 conn(0x7fe790036610 msgr2=0x7fe790005eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:51.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.092+0000 7f16a6a78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:51.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.092+0000 7f16a6a78640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:51.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.092+0000 7f16a6a78640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:51.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.092+0000 7fe79d020640 1 -- 172.21.15.7:0/1182495510 shutdown_connections 2024-03-28T16:15:51.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.092+0000 7fe79d020640 1 -- 172.21.15.7:0/1182495510 wait complete. 2024-03-28T16:15:51.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.093+0000 7fe79d020640 10 monclient: build_initial_monmap 2024-03-28T16:15:51.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.093+0000 7fe79d020640 10 monclient: monmap: 2024-03-28T16:15:51.113 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:51.113 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:51.113 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:51.113 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:51.113 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:51.113 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.113 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:51.114 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:51.114 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:51.114 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.093+0000 7fe79d020640 1 librados: starting msgr at 2024-03-28T16:15:51.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.093+0000 7fe79d020640 1 librados: starting objecter 2024-03-28T16:15:51.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.093+0000 7f16a5a76640 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2024-03-28T16:15:51.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.093+0000 7f16a5a76640 1 -- 172.21.15.7:0/2557141569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16a013ee50 msgr2=0x7f16a00b02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.093+0000 7f16a5a76640 1 --2- 172.21.15.7:0/2557141569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16a013ee50 0x7f16a00b02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.093+0000 7f16a5a76640 1 -- 172.21.15.7:0/2557141569 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16a0071480 msgr2=0x7f16a013fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.093+0000 7f16a5a76640 1 --2- 172.21.15.7:0/2557141569 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16a0071480 0x7f16a013fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.093+0000 7f16a5a76640 10 monclient: _finish_hunting 0 2024-03-28T16:15:51.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.093+0000 7f16a5a76640 1 monclient: found mon.noname-c 2024-03-28T16:15:51.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.093+0000 7f16a5a76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:51.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.093+0000 7fe79d020640 1 Processor -- start 2024-03-28T16:15:51.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.093+0000 7f16a5a76640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-03-28T16:15:51.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.093+0000 7f16a5a76640 1 -- 172.21.15.7:0/2557141569 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1698009a70 con 0x7f16a00b32e0 2024-03-28T16:15:51.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.094+0000 7fe79d020640 1 -- start start 2024-03-28T16:15:51.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.094+0000 7fe79d020640 1 librados: setting wanted keys 2024-03-28T16:15:51.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.094+0000 7fe79d020640 1 librados: calling monclient init 2024-03-28T16:15:51.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.094+0000 7fe79d020640 10 monclient: init 2024-03-28T16:15:51.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.094+0000 7fe79d020640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:51.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.094+0000 7fe79d020640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-28T16:15:51.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.094+0000 7fe79d020640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe79002e1d0 0x7fe7900caae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.094+0000 7fe79d020640 10 monclient(hunting): picked mon.noname-b con 0x7fe79002e1d0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:51.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.094+0000 7fe79d020640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe79002ec10 0x7fe7900cb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.094+0000 7fe79d020640 10 monclient(hunting): picked mon.noname-a con 0x7fe79002ec10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:51.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.094+0000 7fe79d020640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe790032ec0 0x7fe7900b7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.094+0000 7fe79d020640 10 monclient(hunting): picked mon.noname-c con 0x7fe790032ec0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:51.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.094+0000 7fe79d020640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.094+0000 7fe79d020640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe790029e60 con 0x7fe79002ec10 2024-03-28T16:15:51.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.094+0000 7fe79d020640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.094+0000 7fe79d020640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79002a140 con 0x7fe79002e1d0 2024-03-28T16:15:51.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.094+0000 7fe79d020640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.094+0000 7fe79d020640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7900cb020 con 0x7fe790032ec0 2024-03-28T16:15:51.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.094+0000 7fe79d020640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:51.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.094+0000 7fe79d020640 10 monclient(hunting): authenticate will time out at 1418.236624s 2024-03-28T16:15:51.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.097+0000 7f3fcffff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3fd00a86e0 0x7f3fd0144f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.097+0000 7fe79c81f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe790032ec0 0x7fe7900b7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.097+0000 7f3fcffff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3fd00a86e0 0x7f3fd0144f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60240/0 (socket says 172.21.15.7:60240) 2024-03-28T16:15:51.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.097+0000 7f3fcffff640 1 -- 172.21.15.7:0/3698726909 learned_addr learned my addr 172.21.15.7:0/3698726909 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:51.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.097+0000 7f3fcffff640 10 monclient(hunting): get_auth_request con 0x7f3fd00a86e0 auth_method 0 2024-03-28T16:15:51.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.097+0000 7f3fcffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:51.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.097+0000 7f3fcffff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:51.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.097+0000 7f3fcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:51.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.098+0000 7f3fcffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:51.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.098+0000 7f3fcffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:51.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.098+0000 7f3fcffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:51.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.098+0000 7f95b3fff640 1 --2- 172.21.15.7:0/4240112885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95ac035640 0x7f95ac0cc0c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f95a8009870 tx=0x7f95a802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfcdde9aea6f0f73 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.099+0000 7fe79c81f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe790032ec0 0x7fe7900b7610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60252/0 (socket says 172.21.15.7:60252) 2024-03-28T16:15:51.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.099+0000 7fe79c81f640 1 -- 172.21.15.7:0/3116337946 learned_addr learned my addr 172.21.15.7:0/3116337946 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:51.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.099+0000 7fe79c81f640 10 monclient(hunting): get_auth_request con 0x7fe790032ec0 auth_method 0 2024-03-28T16:15:51.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.099+0000 7fe79c81f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:51.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.099+0000 7fe79c81f640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:51.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.099+0000 7fe79c81f640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:51.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.101+0000 7f3fcffff640 10 monclient(hunting): handle_auth_done global_id 4289 payload 274 2024-03-28T16:15:51.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.101+0000 7f3fcffff640 1 -- 172.21.15.7:0/3698726909 >> v2:172.21.15.83:3300/0 conn(0x7f3fd00b32e0 msgr2=0x7f3fd0144a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:51.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.101+0000 7f3fcffff640 1 --2- 172.21.15.7:0/3698726909 >> v2:172.21.15.83:3300/0 conn(0x7f3fd00b32e0 0x7f3fd0144a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.101+0000 7f3fcffff640 1 -- 172.21.15.7:0/3698726909 >> v1:172.21.15.7:6789/0 conn(0x7f3fd0071480 legacy=0x7f3fd013e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:51.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.101+0000 7f3fcffff640 10 monclient: _finish_hunting 0 2024-03-28T16:15:51.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.101+0000 7f3fcffff640 1 monclient: found mon.noname-b 2024-03-28T16:15:51.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.101+0000 7f3fcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:51.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.101+0000 7f3fcffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-03-28T16:15:51.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.101+0000 7f3fcffff640 1 -- 172.21.15.7:0/3698726909 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd0072140 con 0x7f3fd00a86e0 2024-03-28T16:15:51.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.103+0000 7f9d64e49640 1 -- 172.21.15.7:0/764467886 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c031830 con 0x7f9d600b32e0 2024-03-28T16:15:51.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.103+0000 7f9d64e49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.103+0000 7f9d64e49640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.103+0000 7f9d64e49640 10 monclient: dump: 2024-03-28T16:15:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.124 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.103+0000 7f9d64e49640 10 monclient: _finish_auth 0 2024-03-28T16:15:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.103+0000 7f9d64e49640 10 monclient: _check_auth_tickets 2024-03-28T16:15:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.103+0000 7f9d64e49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.103+0000 7f9d64e49640 1 -- 172.21.15.7:0/764467886 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5c031990 con 0x7f9d600b32e0 2024-03-28T16:15:51.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.103+0000 7f9d680af640 5 monclient: authenticate success, global_id 4336 2024-03-28T16:15:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.103+0000 7f9d680af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.103+0000 7f9d64e49640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.103+0000 7f9d680af640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.103+0000 7f9d680af640 4 set_mon_vals no callback set 2024-03-28T16:15:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.103+0000 7f9d64e49640 1 -- 172.21.15.7:0/764467886 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c031d60 con 0x7f9d600b32e0 2024-03-28T16:15:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.103+0000 7f9d64e49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.103+0000 7f9d64e49640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.103+0000 7f9d64e49640 10 monclient: dump: 2024-03-28T16:15:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.125 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.126 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.126 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.103+0000 7f16a5a76640 1 --2- 172.21.15.7:0/2557141569 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16a00b32e0 0x7f16a00afd70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f169000b4d0 tx=0x7f169000b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.103+0000 7f1696ffd640 10 client.?.objecter ms_handle_connect 0x7f16a00b32e0 2024-03-28T16:15:51.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.103+0000 7f1696ffd640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:51.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.104+0000 7f9d680af640 10 monclient: shutdown 2024-03-28T16:15:51.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.104+0000 7f9d680af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:51.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.104+0000 7f9d680af640 1 -- 172.21.15.7:0/764467886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d600b32e0 msgr2=0x7f9d60144a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.104+0000 7f9d680af640 1 --2- 172.21.15.7:0/764467886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d600b32e0 0x7f9d60144a00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9d60071ac0 tx=0x7f9d5c02b8d0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.104+0000 7f9d680af640 1 -- 172.21.15.7:0/764467886 shutdown_connections 2024-03-28T16:15:51.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.104+0000 7f9d680af640 1 --2- 172.21.15.7:0/764467886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d600b32e0 0x7f9d60144a00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.104+0000 7f9d680af640 1 --2- 172.21.15.7:0/764467886 >> v2:172.21.15.7:3300/0 conn(0x7f9d60071480 0x7f9d6013e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.104+0000 7f9d680af640 1 -- 172.21.15.7:0/764467886 >> 172.21.15.7:0/764467886 conn(0x7f9d600b70e0 msgr2=0x7f9d600b74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:51.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.104+0000 7f9d680af640 1 -- 172.21.15.7:0/764467886 shutdown_connections 2024-03-28T16:15:51.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.104+0000 7f9d680af640 1 -- 172.21.15.7:0/764467886 wait complete. 2024-03-28T16:15:51.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.104+0000 7f9d680af640 10 monclient: build_initial_monmap 2024-03-28T16:15:51.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.105+0000 7f9d680af640 10 monclient: monmap: 2024-03-28T16:15:51.128 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:51.128 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:51.128 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:51.128 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:51.128 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:51.128 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.128 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:51.128 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:51.128 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:51.128 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.105+0000 7fe79c81f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:51.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.105+0000 7fe79c81f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:51.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.105+0000 7fe79c81f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:51.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.105+0000 7f9d680af640 1 librados: starting msgr at 2024-03-28T16:15:51.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.105+0000 7f9d680af640 1 librados: starting objecter 2024-03-28T16:15:51.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.105+0000 7f9d680af640 1 Processor -- start 2024-03-28T16:15:51.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.105+0000 7f9d680af640 1 -- start start 2024-03-28T16:15:51.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.105+0000 7f9d680af640 1 librados: setting wanted keys 2024-03-28T16:15:51.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.105+0000 7f9d680af640 1 librados: calling monclient init 2024-03-28T16:15:51.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.105+0000 7f9d680af640 10 monclient: init 2024-03-28T16:15:51.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.106+0000 7f9d680af640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:51.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.106+0000 7f9d680af640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-28T16:15:51.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.106+0000 7f9d680af640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d60071480 0x7f9d601d6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.106+0000 7f9d680af640 10 monclient(hunting): picked mon.noname-a con 0x7f9d60071480 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:51.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.106+0000 7f9d680af640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9d600b32e0 0x7f9d60143030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.106+0000 7f9d680af640 10 monclient(hunting): picked mon.noname-c con 0x7f9d600b32e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:51.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.106+0000 7f9d680af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d6013ee50 0x7f9d6013ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.106+0000 7f9d680af640 10 monclient(hunting): picked mon.noname-b con 0x7f9d6013ee50 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:51.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.106+0000 7f9d680af640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.106+0000 7f9d680af640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6013d890 con 0x7f9d60071480 2024-03-28T16:15:51.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.106+0000 7f9d680af640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.106+0000 7f9d680af640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6013e2f0 con 0x7f9d6013ee50 2024-03-28T16:15:51.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.106+0000 7f9d680af640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.106+0000 7f9d680af640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d60142d30 con 0x7f9d600b32e0 2024-03-28T16:15:51.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.106+0000 7f9d680af640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:51.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.106+0000 7f9d680af640 10 monclient(hunting): authenticate will time out at 1418.248130s 2024-03-28T16:15:51.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.106+0000 7fe797fff640 1 --2- 172.21.15.7:0/3116337946 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe79002e1d0 0x7fe7900caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.107+0000 7f95b37fe640 1 -- 172.21.15.7:0/4240112885 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a8031870 con 0x7f95ac035640 2024-03-28T16:15:51.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.107+0000 7f95b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.107+0000 7f95b37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:51.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.107+0000 7f95b37fe640 10 monclient: dump: 2024-03-28T16:15:51.132 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.132 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.132 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.132 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.133 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.133 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.133 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.133 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.133 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.133 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.107+0000 7f95b37fe640 10 monclient: _finish_auth 0 2024-03-28T16:15:51.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.107+0000 7f95b37fe640 10 monclient: _check_auth_tickets 2024-03-28T16:15:51.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.107+0000 7f95b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:51.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.107+0000 7f95b99df640 5 monclient: authenticate success, global_id 4348 2024-03-28T16:15:51.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.107+0000 7f95b99df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:51.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.107+0000 7f95b37fe640 1 -- 172.21.15.7:0/4240112885 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a80319d0 con 0x7f95ac035640 2024-03-28T16:15:51.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.107+0000 7f95b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:51.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.107+0000 7f95b99df640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:51.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.107+0000 7f95b99df640 4 set_mon_vals no callback set 2024-03-28T16:15:51.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.108+0000 7f95b37fe640 1 -- 172.21.15.7:0/4240112885 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a8031d60 con 0x7f95ac035640 2024-03-28T16:15:51.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.108+0000 7f95b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.108+0000 7f95b37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:51.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.108+0000 7f95b37fe640 10 monclient: dump: 2024-03-28T16:15:51.134 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.134 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.134 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.134 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.134 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.134 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.134 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.134 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.134 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.134 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.108+0000 7f95b99df640 10 monclient: shutdown 2024-03-28T16:15:51.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.108+0000 7f95b99df640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:51.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.108+0000 7f95b99df640 1 -- 172.21.15.7:0/4240112885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95ac035640 msgr2=0x7f95ac0cc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.108+0000 7f95b99df640 1 --2- 172.21.15.7:0/4240112885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95ac035640 0x7f95ac0cc0c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f95a8009870 tx=0x7f95a802b910 comp rx=0 tx=0).stop 2024-03-28T16:15:51.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.108+0000 7fe797fff640 10 monclient(hunting): get_auth_request con 0x7fe79002e1d0 auth_method 0 2024-03-28T16:15:51.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.108+0000 7fe797fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:51.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.108+0000 7fe797fff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:51.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.108+0000 7fe797fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:51.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.108+0000 7f95b99df640 1 -- 172.21.15.7:0/4240112885 shutdown_connections 2024-03-28T16:15:51.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.108+0000 7f95b99df640 1 --2- 172.21.15.7:0/4240112885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95ac035640 0x7f95ac0cc0c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.108+0000 7f95b99df640 1 -- 172.21.15.7:0/4240112885 >> 172.21.15.7:0/4240112885 conn(0x7f95ac0065d0 msgr2=0x7f95ac0069e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:51.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.108+0000 7f95b99df640 1 -- 172.21.15.7:0/4240112885 shutdown_connections 2024-03-28T16:15:51.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.108+0000 7f95b99df640 1 -- 172.21.15.7:0/4240112885 wait complete. 2024-03-28T16:15:51.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.109+0000 7f95b99df640 10 monclient: build_initial_monmap 2024-03-28T16:15:51.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.109+0000 7f95b99df640 10 monclient: monmap: 2024-03-28T16:15:51.136 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:51.136 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:51.136 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:51.136 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:51.136 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:51.137 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.137 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:51.137 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:51.137 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:51.137 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.109+0000 7fe79c81f640 10 monclient(hunting): handle_auth_done global_id 4292 payload 824 2024-03-28T16:15:51.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.109+0000 7fe79c81f640 1 -- 172.21.15.7:0/3116337946 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe79002e1d0 msgr2=0x7fe7900caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.109+0000 7fe79c81f640 1 --2- 172.21.15.7:0/3116337946 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe79002e1d0 0x7fe7900caae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.109+0000 7fe79c81f640 1 -- 172.21.15.7:0/3116337946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe79002ec10 msgr2=0x7fe7900cb3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.109+0000 7fe79c81f640 1 --2- 172.21.15.7:0/3116337946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe79002ec10 0x7fe7900cb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.109+0000 7fe79c81f640 10 monclient: _finish_hunting 0 2024-03-28T16:15:51.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.109+0000 7fe79c81f640 1 monclient: found mon.noname-c 2024-03-28T16:15:51.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.109+0000 7fe79c81f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:51.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.109+0000 7fe79c81f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-03-28T16:15:51.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.109+0000 7fe79c81f640 1 -- 172.21.15.7:0/3116337946 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7840094d0 con 0x7fe790032ec0 2024-03-28T16:15:51.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.109+0000 7f95b99df640 1 librados: starting msgr at 2024-03-28T16:15:51.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.109+0000 7f95b99df640 1 librados: starting objecter 2024-03-28T16:15:51.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.110+0000 7f95b99df640 1 Processor -- start 2024-03-28T16:15:51.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.110+0000 7f95b99df640 1 -- start start 2024-03-28T16:15:51.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.110+0000 7f95b99df640 1 librados: setting wanted keys 2024-03-28T16:15:51.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.110+0000 7f95b99df640 1 librados: calling monclient init 2024-03-28T16:15:51.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.110+0000 7f95b99df640 10 monclient: init 2024-03-28T16:15:51.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.111+0000 7f95b99df640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:51.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.111+0000 7f95b99df640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-28T16:15:51.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.111+0000 7f95b99df640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95ac035640 0x7f95ac0c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.111+0000 7f95b99df640 10 monclient(hunting): picked mon.noname-b con 0x7f95ac035640 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:51.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.111+0000 7f95b99df640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95ac038fe0 0x7f95ac0ca070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.111+0000 7f95b99df640 10 monclient(hunting): picked mon.noname-a con 0x7f95ac038fe0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:51.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.111+0000 7f95b99df640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95ac0cc600 0x7f95ac0ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.111+0000 7f95b99df640 10 monclient(hunting): picked mon.noname-c con 0x7f95ac0cc600 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:51.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.111+0000 7f95b99df640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.111+0000 7f95b99df640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac0cbe70 con 0x7f95ac038fe0 2024-03-28T16:15:51.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.111+0000 7f95b99df640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.111+0000 7f95b99df640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac0c8ac0 con 0x7f95ac035640 2024-03-28T16:15:51.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.111+0000 7f95b99df640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.111+0000 7f95b99df640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac0c8300 con 0x7f95ac0cc600 2024-03-28T16:15:51.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.111+0000 7f95b99df640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:51.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.111+0000 7f95b99df640 10 monclient(hunting): authenticate will time out at 1418.253035s 2024-03-28T16:15:51.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.112+0000 7f9d6664c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d6013ee50 0x7f9d6013ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.112+0000 7f9d6664c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d6013ee50 0x7f9d6013ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:58150/0 (socket says 172.21.15.7:58150) 2024-03-28T16:15:51.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.112+0000 7f9d6664c640 1 -- 172.21.15.7:0/2093165412 learned_addr learned my addr 172.21.15.7:0/2093165412 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:51.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.112+0000 7f9d6664c640 10 monclient(hunting): get_auth_request con 0x7f9d6013ee50 auth_method 0 2024-03-28T16:15:51.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.112+0000 7f9d6664c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:51.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.112+0000 7f9d6664c640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:51.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.112+0000 7f9d6664c640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:51.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.112+0000 7f95b89dd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95ac035640 0x7f95ac0c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.113+0000 7f9d6664c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:51.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.113+0000 7f9d6664c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:51.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.113+0000 7f9d6664c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:51.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.113+0000 7f3fcffff640 1 --2- 172.21.15.7:0/3698726909 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3fd00a86e0 0x7f3fd0144f40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8009e50 tx=0x7f3fc802bdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d31d4b3d83fba151 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.113+0000 7f95b89dd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95ac035640 0x7f95ac0c7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:58162/0 (socket says 172.21.15.7:58162) 2024-03-28T16:15:51.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.113+0000 7f95b89dd640 1 -- 172.21.15.7:0/3321025491 learned_addr learned my addr 172.21.15.7:0/3321025491 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:51.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.113+0000 7f95b89dd640 10 monclient(hunting): get_auth_request con 0x7f95ac035640 auth_method 0 2024-03-28T16:15:51.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.113+0000 7f95b89dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:51.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.113+0000 7f95b89dd640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:51.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.113+0000 7f95b89dd640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:51.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.114+0000 7f1b05b9c640 10 monclient: get_monmap_and_config 2024-03-28T16:15:51.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.114+0000 7f1b05b9c640 10 monclient: build_initial_monmap 2024-03-28T16:15:51.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.114+0000 7f1b05b9c640 10 monclient: monmap: 2024-03-28T16:15:51.144 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:51.115669+0000 2024-03-28T16:15:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:51.115669+0000 2024-03-28T16:15:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:51.145 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.114+0000 7f1b05b9c640 1 Processor -- start 2024-03-28T16:15:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.115+0000 7f1b05b9c640 1 -- start start 2024-03-28T16:15:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.115+0000 7f1b05b9c640 10 monclient: init 2024-03-28T16:15:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.115+0000 7f1b05b9c640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:51.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.115+0000 7f1b05b9c640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-03-28T16:15:51.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.115+0000 7f1b05b9c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1b00071480 0x7f1b0013e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f1b05b9c640 10 monclient(hunting): picked mon.noname-b con 0x7f1b00071480 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:51.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f1b05b9c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1b000b32e0 addr v1:172.21.15.83:6789/0 2024-03-28T16:15:51.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f1b05b9c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1b0013ee50 0x7f1b00145110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f1b05b9c640 10 monclient(hunting): picked mon.noname-c con 0x7f1b0013ee50 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:51.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f1b05b9c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b000a8c90 con 0x7f1b000b32e0 2024-03-28T16:15:51.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f1b05b9c640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f1b05b9c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b00071860 con 0x7f1b0013ee50 2024-03-28T16:15:51.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f9d6664c640 10 monclient(hunting): handle_auth_done global_id 4360 payload 824 2024-03-28T16:15:51.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f9d6664c640 1 -- 172.21.15.7:0/2093165412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9d600b32e0 msgr2=0x7f9d60143030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f9d6664c640 1 --2- 172.21.15.7:0/2093165412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9d600b32e0 0x7f9d60143030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f9d6664c640 1 -- 172.21.15.7:0/2093165412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d60071480 msgr2=0x7f9d601d6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f1b05b9c640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f9d6664c640 1 --2- 172.21.15.7:0/2093165412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d60071480 0x7f9d601d6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f9d6664c640 10 monclient: _finish_hunting 0 2024-03-28T16:15:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f9d6664c640 1 monclient: found mon.noname-b 2024-03-28T16:15:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f9d6664c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f9d6664c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:15:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f9d6664c640 1 -- 172.21.15.7:0/2093165412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5c0094d0 con 0x7f9d6013ee50 2024-03-28T16:15:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f1b05b9c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b00145650 con 0x7f1b00071480 2024-03-28T16:15:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f1b05b9c640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f1b05b9c640 10 monclient(hunting): authenticate will time out at 1418.258254s 2024-03-28T16:15:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f95b89dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f95b89dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:51.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.116+0000 7f95b89dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.117+0000 7ffbf9046640 10 monclient: get_monmap_and_config 2024-03-28T16:15:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.117+0000 7ffbf9046640 10 monclient: build_initial_monmap 2024-03-28T16:15:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.117+0000 7ffbf9046640 10 monclient: monmap: 2024-03-28T16:15:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:51.119431+0000 2024-03-28T16:15:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:51.119431+0000 2024-03-28T16:15:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:51.149 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:51.150 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.118+0000 7ffbf9046640 1 Processor -- start 2024-03-28T16:15:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.118+0000 7ffbf9046640 1 -- start start 2024-03-28T16:15:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.118+0000 7ffbf9046640 10 monclient: init 2024-03-28T16:15:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.119+0000 7fe79c81f640 1 --2- 172.21.15.7:0/3116337946 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe790032ec0 0x7fe7900b7610 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe7840090b0 tx=0x7fe7840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.119+0000 7fe794ff9640 10 client.?.objecter ms_handle_connect 0x7fe790032ec0 2024-03-28T16:15:51.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.119+0000 7fe794ff9640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.119+0000 7ffbf9046640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.119+0000 7ffbf9046640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-03-28T16:15:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.119+0000 7ffbf9046640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7ffbf4071480 0x7ffbf413e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.119+0000 7ffbf9046640 10 monclient(hunting): picked mon.noname-c con 0x7ffbf4071480 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.119+0000 7ffbf9046640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffbf40b32e0 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.119+0000 7ffbf9046640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffbf413ee50 0x7ffbf4145110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.119+0000 7ffbf9046640 10 monclient(hunting): picked mon.noname-a con 0x7ffbf413ee50 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.119+0000 7ffbf9046640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbf40a8c90 con 0x7ffbf40b32e0 2024-03-28T16:15:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.119+0000 7ffbf9046640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.119+0000 7ffbf9046640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbf4071860 con 0x7ffbf413ee50 2024-03-28T16:15:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.119+0000 7ffbf9046640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.119+0000 7ffbf9046640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbf4145650 con 0x7ffbf4071480 2024-03-28T16:15:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.119+0000 7ffbf9046640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.119+0000 7ffbf9046640 10 monclient(hunting): authenticate will time out at 1418.261478s 2024-03-28T16:15:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.120+0000 7f95b3fff640 1 --2- 172.21.15.7:0/3321025491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95ac038fe0 0x7f95ac0ca070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.120+0000 7f95b89dd640 10 monclient(hunting): handle_auth_done global_id 4363 payload 824 2024-03-28T16:15:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.120+0000 7f95b89dd640 1 -- 172.21.15.7:0/3321025491 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95ac0cc600 msgr2=0x7f95ac0ca5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.120+0000 7f95b89dd640 1 --2- 172.21.15.7:0/3321025491 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95ac0cc600 0x7f95ac0ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.120+0000 7f95b89dd640 1 -- 172.21.15.7:0/3321025491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95ac038fe0 msgr2=0x7f95ac0ca070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.120+0000 7f95b89dd640 1 --2- 172.21.15.7:0/3321025491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95ac038fe0 0x7f95ac0ca070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.120+0000 7f95b89dd640 10 monclient: _finish_hunting 0 2024-03-28T16:15:51.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.120+0000 7f95b89dd640 1 monclient: found mon.noname-b 2024-03-28T16:15:51.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.120+0000 7f95b89dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:51.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.120+0000 7f95b89dd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:15:51.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.120+0000 7f95b89dd640 1 -- 172.21.15.7:0/3321025491 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a80094d0 con 0x7f95ac035640 2024-03-28T16:15:51.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.122+0000 7f9d6664c640 1 --2- 172.21.15.7:0/2093165412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d6013ee50 0x7f9d6013ddb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800c730 tx=0x7f9d5800cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.122+0000 7f9d567fc640 10 client.?.objecter ms_handle_connect 0x7f9d6013ee50 2024-03-28T16:15:51.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.122+0000 7f9d567fc640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:51.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.123+0000 7f1aff7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1b00071480 0x7f1b0013e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.124+0000 7ffbf359e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffbf413ee50 0x7ffbf4145110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.124+0000 7ffbf359e640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ffbf413ee50 0x7ffbf4145110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37994/0 (socket says 172.21.15.7:37994) 2024-03-28T16:15:51.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.124+0000 7ffbf359e640 1 -- 172.21.15.7:0/2354539416 learned_addr learned my addr 172.21.15.7:0/2354539416 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:51.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.124+0000 7ffbf359e640 10 monclient(hunting): get_auth_request con 0x7ffbf413ee50 auth_method 0 2024-03-28T16:15:51.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.124+0000 7ffbf359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:51.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.124+0000 7ffbf359e640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.124+0000 7ffbf359e640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.125+0000 7f1aff7fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1b00071480 0x7f1b0013e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60290/0 (socket says 172.21.15.7:60290) 2024-03-28T16:15:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.125+0000 7f1aff7fe640 1 -- 172.21.15.7:0/1829584434 learned_addr learned my addr 172.21.15.7:0/1829584434 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.125+0000 7f1aff7fe640 10 monclient(hunting): get_auth_request con 0x7f1b00071480 auth_method 0 2024-03-28T16:15:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.125+0000 7f1aff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.125+0000 7f1aff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.125+0000 7f1aff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.125+0000 7ffbf359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:51.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.125+0000 7ffbf359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:51.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.125+0000 7ffbf359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:51.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.125+0000 7f1696ffd640 1 -- 172.21.15.7:0/2557141569 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1690010870 con 0x7f16a00b32e0 2024-03-28T16:15:51.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.125+0000 7f1696ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.125+0000 7f1696ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:51.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.125+0000 7f1696ffd640 10 monclient: dump: 2024-03-28T16:15:51.157 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.158 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.158 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.158 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.158 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.158 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.158 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.158 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.158 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.158 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.125+0000 7f1696ffd640 10 monclient: _finish_auth 0 2024-03-28T16:15:51.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.125+0000 7f1696ffd640 10 monclient: _check_auth_tickets 2024-03-28T16:15:51.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.125+0000 7f1696ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:51.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.125+0000 7f1696ffd640 1 -- 172.21.15.7:0/2557141569 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16900109d0 con 0x7f16a00b32e0 2024-03-28T16:15:51.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.125+0000 7f16a84db640 5 monclient: authenticate success, global_id 4286 2024-03-28T16:15:51.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.126+0000 7f1696ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.126+0000 7f1696ffd640 1 -- 172.21.15.7:0/2557141569 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1690010d80 con 0x7f16a00b32e0 2024-03-28T16:15:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.126+0000 7f1696ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.126+0000 7f1696ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:15:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.126+0000 7f1696ffd640 10 monclient: dump: 2024-03-28T16:15:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.159 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.126+0000 7f16a4a74640 4 set_mon_vals no callback set 2024-03-28T16:15:51.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.126+0000 7f16a84db640 10 monclient: _renew_subs 2024-03-28T16:15:51.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.126+0000 7f16a84db640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:51.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.126+0000 7f16a84db640 1 -- 172.21.15.7:0/2557141569 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16a0140a30 con 0x7f16a00b32e0 2024-03-28T16:15:51.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.126+0000 7f16a84db640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:51.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.126+0000 7f16a84db640 10 monclient: _renew_subs 2024-03-28T16:15:51.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.126+0000 7f16a84db640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:51.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.126+0000 7ffbf2d9d640 1 --2- 172.21.15.7:0/2354539416 >> v2:172.21.15.83:3300/0 conn(0x7ffbf4071480 0x7ffbf413e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.126+0000 7f16a84db640 1 -- 172.21.15.7:0/2557141569 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16a0144610 con 0x7f16a00b32e0 2024-03-28T16:15:51.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.126+0000 7f16a84db640 1 librados: init done 2024-03-28T16:15:51.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.126+0000 7f1afffff640 1 --2- 172.21.15.7:0/1829584434 >> v2:172.21.15.83:3300/0 conn(0x7f1b0013ee50 0x7f1b00145110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.126+0000 7f95b89dd640 1 --2- 172.21.15.7:0/3321025491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95ac035640 0x7f95ac0c7dc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f95a000c410 tx=0x7f95a000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.126+0000 7f95b17fa640 10 client.?.objecter ms_handle_connect 0x7f95ac035640 2024-03-28T16:15:51.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.126+0000 7f95b17fa640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:51.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.127+0000 7f1afffff640 10 monclient(hunting): get_auth_request con 0x7f1b0013ee50 auth_method 0 2024-03-28T16:15:51.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.127+0000 7f1afffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:51.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.127+0000 7f1afffff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:51.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.127+0000 7f1afffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:51.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.127+0000 7f16a84db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.127+0000 7f1aff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:51.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.127+0000 7f1aff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:51.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.127+0000 7f1aff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:51.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.127+0000 7f16a84db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.127+0000 7f16a84db640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:51.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.127+0000 7f16a84db640 1 -- 172.21.15.7:0/2557141569 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16a0140c20 con 0x7f16a00b32e0 2024-03-28T16:15:51.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.127+0000 7ffbf2d9d640 10 monclient(hunting): get_auth_request con 0x7ffbf4071480 auth_method 0 2024-03-28T16:15:51.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.127+0000 7ffbf2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:51.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.127+0000 7ffbf2d9d640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:51.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.127+0000 7ffbf2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:51.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.128+0000 7f1afffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:51.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.128+0000 7f1afffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:51.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.128+0000 7f1afffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:51.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.128+0000 7ffbf359e640 10 monclient(hunting): handle_auth_done global_id 4329 payload 274 2024-03-28T16:15:51.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.128+0000 7ffbf359e640 1 -- 172.21.15.7:0/2354539416 >> v2:172.21.15.83:3300/0 conn(0x7ffbf4071480 msgr2=0x7ffbf413e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:51.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.128+0000 7ffbf359e640 1 --2- 172.21.15.7:0/2354539416 >> v2:172.21.15.83:3300/0 conn(0x7ffbf4071480 0x7ffbf413e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.128+0000 7ffbf359e640 1 -- 172.21.15.7:0/2354539416 >> v1:172.21.15.7:6789/0 conn(0x7ffbf40b32e0 legacy=0x7ffbf4144a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.128+0000 7ffbf359e640 10 monclient: _finish_hunting 0 2024-03-28T16:15:51.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.128+0000 7ffbf359e640 1 monclient: found mon.noname-a 2024-03-28T16:15:51.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.128+0000 7ffbf359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:51.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.128+0000 7ffbf359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:15:51.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.128+0000 7ffbf359e640 1 -- 172.21.15.7:0/2354539416 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbf4072140 con 0x7ffbf413ee50 2024-03-28T16:15:51.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f1aff7fe640 10 monclient(hunting): handle_auth_done global_id 4295 payload 274 2024-03-28T16:15:51.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f1aff7fe640 1 -- 172.21.15.7:0/1829584434 >> v2:172.21.15.83:3300/0 conn(0x7f1b0013ee50 msgr2=0x7f1b00145110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:51.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f1aff7fe640 1 --2- 172.21.15.7:0/1829584434 >> v2:172.21.15.83:3300/0 conn(0x7f1b0013ee50 0x7f1b00145110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f1aff7fe640 1 -- 172.21.15.7:0/1829584434 >> v1:172.21.15.83:6789/0 conn(0x7f1b000b32e0 legacy=0x7f1b00144a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:51.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f1aff7fe640 10 monclient: _finish_hunting 0 2024-03-28T16:15:51.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f1aff7fe640 1 monclient: found mon.noname-b 2024-03-28T16:15:51.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f1aff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:51.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f1aff7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-03-28T16:15:51.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f1aff7fe640 1 -- 172.21.15.7:0/1829584434 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b00072140 con 0x7f1b00071480 2024-03-28T16:15:51.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f9d567fc640 1 -- 172.21.15.7:0/2093165412 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d58006d50 con 0x7f9d6013ee50 2024-03-28T16:15:51.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f9d567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f9d567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:51.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f9d567fc640 10 monclient: dump: 2024-03-28T16:15:51.168 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.168 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.168 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.168 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.168 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.168 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.168 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.168 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.168 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.168 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f9d567fc640 10 monclient: _finish_auth 0 2024-03-28T16:15:51.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f9d567fc640 10 monclient: _check_auth_tickets 2024-03-28T16:15:51.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f9d567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:51.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f9d567fc640 1 -- 172.21.15.7:0/2093165412 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d58003970 con 0x7f9d6013ee50 2024-03-28T16:15:51.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f9d567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:51.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f9d567fc640 1 -- 172.21.15.7:0/2093165412 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d58003d60 con 0x7f9d6013ee50 2024-03-28T16:15:51.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f9d567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f9d567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:51.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f9d567fc640 10 monclient: dump: 2024-03-28T16:15:51.169 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.169 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.169 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.169 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.169 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.169 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.169 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.170 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.170 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.170 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f9d57fff640 4 set_mon_vals no callback set 2024-03-28T16:15:51.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.130+0000 7f9d680af640 5 monclient: authenticate success, global_id 4360 2024-03-28T16:15:51.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.131+0000 7f9d680af640 10 monclient: _renew_subs 2024-03-28T16:15:51.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.131+0000 7f9d680af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:51.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.131+0000 7f9d680af640 1 -- 172.21.15.7:0/2093165412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d601d6f70 con 0x7f9d6013ee50 2024-03-28T16:15:51.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.131+0000 7f9d680af640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:51.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.131+0000 7f9d680af640 10 monclient: _renew_subs 2024-03-28T16:15:51.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.131+0000 7f9d680af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:51.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.131+0000 7f9d680af640 1 -- 172.21.15.7:0/2093165412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d601d7220 con 0x7f9d6013ee50 2024-03-28T16:15:51.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.131+0000 7f9d680af640 1 librados: init done 2024-03-28T16:15:51.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f9d680af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f9d680af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f9d680af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:51.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7ffbf359e640 1 --2- 172.21.15.7:0/2354539416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbf413ee50 0x7ffbf4145110 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffbf40b4a10 tx=0x7ffbec02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb4a3d6da0e7d91a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f9d680af640 1 -- 172.21.15.7:0/2093165412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d6013d670 con 0x7f9d6013ee50 2024-03-28T16:15:51.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f3fce7fc640 1 -- 172.21.15.7:0/3698726909 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc802c960 con 0x7f3fd00a86e0 2024-03-28T16:15:51.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f3fce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f3fce7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:51.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f3fce7fc640 10 monclient: dump: 2024-03-28T16:15:51.172 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.172 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.172 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.172 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.172 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.172 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.172 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.172 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.172 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.172 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f3fce7fc640 10 monclient: _finish_auth 0 2024-03-28T16:15:51.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f3fce7fc640 10 monclient: _check_auth_tickets 2024-03-28T16:15:51.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f3fce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:51.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f3fce7fc640 1 -- 172.21.15.7:0/3698726909 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc802cac0 con 0x7f3fd00a86e0 2024-03-28T16:15:51.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f3fce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:51.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f3fce7fc640 1 -- 172.21.15.7:0/3698726909 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc802ce50 con 0x7f3fd00a86e0 2024-03-28T16:15:51.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f3fce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f3fce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:15:51.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f3fce7fc640 10 monclient: dump: 2024-03-28T16:15:51.173 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.173 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.173 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.173 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.173 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.173 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.174 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.174 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.174 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.174 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f3fd5f37640 5 monclient: authenticate success, global_id 4289 2024-03-28T16:15:51.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f3fd5f37640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:51.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f3fd5f37640 4 set_mon_vals no callback set 2024-03-28T16:15:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b17fa640 1 -- 172.21.15.7:0/3321025491 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a00035f0 con 0x7f95ac035640 2024-03-28T16:15:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b17fa640 10 monclient: dump: 2024-03-28T16:15:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.176 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b17fa640 10 monclient: _finish_auth 0 2024-03-28T16:15:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b17fa640 10 monclient: _check_auth_tickets 2024-03-28T16:15:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b17fa640 1 -- 172.21.15.7:0/3321025491 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a0003750 con 0x7f95ac035640 2024-03-28T16:15:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b17fa640 1 -- 172.21.15.7:0/3321025491 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a0003b40 con 0x7f95ac035640 2024-03-28T16:15:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b17fa640 10 monclient: dump: 2024-03-28T16:15:51.177 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.177 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.177 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.177 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.177 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.177 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.177 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.177 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.177 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.177 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b99df640 5 monclient: authenticate success, global_id 4363 2024-03-28T16:15:51.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b99df640 10 monclient: _renew_subs 2024-03-28T16:15:51.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b99df640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:51.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b99df640 1 -- 172.21.15.7:0/3321025491 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95ac0d9510 con 0x7f95ac035640 2024-03-28T16:15:51.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b2ffd640 4 set_mon_vals no callback set 2024-03-28T16:15:51.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b99df640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:51.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b99df640 10 monclient: _renew_subs 2024-03-28T16:15:51.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b99df640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:51.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b99df640 1 -- 172.21.15.7:0/3321025491 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ac0be0a0 con 0x7f95ac035640 2024-03-28T16:15:51.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.132+0000 7f95b99df640 1 librados: init done 2024-03-28T16:15:51.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7f3fd5f37640 10 monclient: shutdown 2024-03-28T16:15:51.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7f3fd5f37640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:51.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7f3fd5f37640 1 -- 172.21.15.7:0/3698726909 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3fd00a86e0 msgr2=0x7f3fd0144f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7f3fd5f37640 1 --2- 172.21.15.7:0/3698726909 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3fd00a86e0 0x7f3fd0144f40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8009e50 tx=0x7f3fc802bdc0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe794ff9640 1 -- 172.21.15.7:0/3116337946 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe784032880 con 0x7fe790032ec0 2024-03-28T16:15:51.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe794ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe794ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:51.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe794ff9640 10 monclient: dump: 2024-03-28T16:15:51.179 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.179 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.179 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.180 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe794ff9640 10 monclient: _finish_auth 0 2024-03-28T16:15:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe794ff9640 10 monclient: _check_auth_tickets 2024-03-28T16:15:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe794ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe794ff9640 1 -- 172.21.15.7:0/3116337946 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe784032a00 con 0x7fe790032ec0 2024-03-28T16:15:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe794ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe7967fc640 4 set_mon_vals no callback set 2024-03-28T16:15:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe794ff9640 1 -- 172.21.15.7:0/3116337946 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe784032db0 con 0x7fe790032ec0 2024-03-28T16:15:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe794ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe794ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:15:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe794ff9640 10 monclient: dump: 2024-03-28T16:15:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.181 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe79d020640 5 monclient: authenticate success, global_id 4292 2024-03-28T16:15:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe79d020640 10 monclient: _renew_subs 2024-03-28T16:15:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe79d020640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe79d020640 1 -- 172.21.15.7:0/3116337946 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7900c9e70 con 0x7fe790032ec0 2024-03-28T16:15:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe79d020640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe79d020640 10 monclient: _renew_subs 2024-03-28T16:15:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe79d020640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe79d020640 1 -- 172.21.15.7:0/3116337946 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7900cbb10 con 0x7fe790032ec0 2024-03-28T16:15:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7fe79d020640 1 librados: init done 2024-03-28T16:15:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.133+0000 7f3fd5f37640 1 -- 172.21.15.7:0/3698726909 shutdown_connections 2024-03-28T16:15:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.134+0000 7f3fd5f37640 1 --2- 172.21.15.7:0/3698726909 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3fd00a86e0 0x7f3fd0144f40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.134+0000 7f3fd5f37640 1 --2- 172.21.15.7:0/3698726909 >> v2:172.21.15.83:3300/0 conn(0x7f3fd00b32e0 0x7f3fd0144a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.134+0000 7f3fd5f37640 1 -- 172.21.15.7:0/3698726909 >> 172.21.15.7:0/3698726909 conn(0x7f3fd00b70e0 msgr2=0x7f3fd00b74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:51.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.134+0000 7f3fd5f37640 1 -- 172.21.15.7:0/3698726909 shutdown_connections 2024-03-28T16:15:51.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.134+0000 7f95b99df640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.134+0000 7f95b99df640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.134+0000 7f95b99df640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:51.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.134+0000 7f95b99df640 1 -- 172.21.15.7:0/3321025491 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95ac0cbfe0 con 0x7f95ac035640 2024-03-28T16:15:51.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.134+0000 7f3fd5f37640 1 -- 172.21.15.7:0/3698726909 wait complete. 2024-03-28T16:15:51.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.135+0000 7f3fd5f37640 10 monclient: build_initial_monmap 2024-03-28T16:15:51.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.135+0000 7f3fd5f37640 10 monclient: monmap: 2024-03-28T16:15:51.184 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:51.184 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:51.184 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:51.184 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:51.184 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:51.185 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.185 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:51.185 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:51.185 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:51.185 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.135+0000 7f3fd5f37640 1 librados: starting msgr at 2024-03-28T16:15:51.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.135+0000 7f3fd5f37640 1 librados: starting objecter 2024-03-28T16:15:51.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.135+0000 7fe79d020640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.135+0000 7f3fd5f37640 1 Processor -- start 2024-03-28T16:15:51.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.135+0000 7fe79d020640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.135+0000 7fe79d020640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:51.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.135+0000 7fe79d020640 1 -- 172.21.15.7:0/3116337946 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe790029fa0 con 0x7fe790032ec0 2024-03-28T16:15:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.135+0000 7f3fd5f37640 1 -- start start 2024-03-28T16:15:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.135+0000 7f3fd5f37640 1 librados: setting wanted keys 2024-03-28T16:15:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.135+0000 7f3fd5f37640 1 librados: calling monclient init 2024-03-28T16:15:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.135+0000 7f3fd5f37640 10 monclient: init 2024-03-28T16:15:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.136+0000 7f3fd5f37640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.136+0000 7f3fd5f37640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-28T16:15:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.136+0000 7f3fd5f37640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fd0071480 0x7f3fd01d6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.136+0000 7f3fd5f37640 10 monclient(hunting): picked mon.noname-b con 0x7f3fd0071480 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.136+0000 7f3fd5f37640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3fd00a86e0 0x7f3fd01d70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.136+0000 7f3fd5f37640 10 monclient(hunting): picked mon.noname-c con 0x7f3fd00a86e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.136+0000 7f3fd5f37640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fd00b32e0 0x7f3fd0138380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.136+0000 7f3fd5f37640 10 monclient(hunting): picked mon.noname-a con 0x7f3fd00b32e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.136+0000 7f3fd5f37640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.136+0000 7f3fd5f37640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd013c990 con 0x7f3fd00b32e0 2024-03-28T16:15:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.136+0000 7f3fd5f37640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.136+0000 7f3fd5f37640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd0131b60 con 0x7f3fd0071480 2024-03-28T16:15:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.136+0000 7f3fd5f37640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.136+0000 7f3fd5f37640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd01d6650 con 0x7f3fd00a86e0 2024-03-28T16:15:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.136+0000 7f3fd5f37640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.136+0000 7f3fd5f37640 10 monclient(hunting): authenticate will time out at 1418.278266s 2024-03-28T16:15:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.137+0000 7ffbf1d9b640 1 -- 172.21.15.7:0/2354539416 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbec02c9f0 con 0x7ffbf413ee50 2024-03-28T16:15:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.137+0000 7ffbf1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:15:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf1d9b640 10 monclient: dump: 2024-03-28T16:15:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.189 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf1d9b640 10 monclient: _finish_auth 0 2024-03-28T16:15:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf1d9b640 10 monclient: _check_auth_tickets 2024-03-28T16:15:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf1d9b640 1 -- 172.21.15.7:0/2354539416 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbec02cb50 con 0x7ffbf413ee50 2024-03-28T16:15:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf1d9b640 1 -- 172.21.15.7:0/2354539416 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbec0343f0 con 0x7ffbf413ee50 2024-03-28T16:15:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf9046640 5 monclient: authenticate success, global_id 4329 2024-03-28T16:15:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf9046640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf9046640 4 set_mon_vals no callback set 2024-03-28T16:15:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:15:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf1d9b640 10 monclient: dump: 2024-03-28T16:15:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.191 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7f3fcffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fd00b32e0 0x7f3fd0138380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7f1aff7fe640 1 --2- 172.21.15.7:0/1829584434 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1b00071480 0x7f1b0013e910 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1af0009870 tx=0x7f1af002b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48ed61588d6fbe7c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf9046640 10 monclient: shutdown 2024-03-28T16:15:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf9046640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf9046640 1 -- 172.21.15.7:0/2354539416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbf413ee50 msgr2=0x7ffbf4145110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf9046640 1 --2- 172.21.15.7:0/2354539416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbf413ee50 0x7ffbf4145110 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffbf40b4a10 tx=0x7ffbec02be50 comp rx=0 tx=0).stop 2024-03-28T16:15:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf9046640 1 -- 172.21.15.7:0/2354539416 shutdown_connections 2024-03-28T16:15:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf9046640 1 --2- 172.21.15.7:0/2354539416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbf413ee50 0x7ffbf4145110 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf9046640 1 --2- 172.21.15.7:0/2354539416 >> v2:172.21.15.83:3300/0 conn(0x7ffbf4071480 0x7ffbf413e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf9046640 1 -- 172.21.15.7:0/2354539416 >> 172.21.15.7:0/2354539416 conn(0x7ffbf40b70e0 msgr2=0x7ffbf40b74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:51.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf9046640 1 -- 172.21.15.7:0/2354539416 shutdown_connections 2024-03-28T16:15:51.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.138+0000 7ffbf9046640 1 -- 172.21.15.7:0/2354539416 wait complete. 2024-03-28T16:15:51.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.139+0000 7f3fcffff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fd00b32e0 0x7f3fd0138380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38008/0 (socket says 172.21.15.7:38008) 2024-03-28T16:15:51.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.139+0000 7f3fcffff640 1 -- 172.21.15.7:0/3691783127 learned_addr learned my addr 172.21.15.7:0/3691783127 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:51.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.139+0000 7f3fcffff640 10 monclient(hunting): get_auth_request con 0x7f3fd00b32e0 auth_method 0 2024-03-28T16:15:51.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.139+0000 7f3fcffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:51.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.139+0000 7f3fcffff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:51.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.139+0000 7f3fcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:51.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.139+0000 7ffbf9046640 10 monclient: build_initial_monmap 2024-03-28T16:15:51.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.139+0000 7ffbf9046640 10 monclient: monmap: 2024-03-28T16:15:51.194 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:51.194 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:51.194 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:51.194 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:51.194 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:51.194 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.194 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:51.195 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:51.195 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:51.195 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.140+0000 7ffbf9046640 1 librados: starting msgr at 2024-03-28T16:15:51.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.140+0000 7ffbf9046640 1 librados: starting objecter 2024-03-28T16:15:51.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.140+0000 7f3fcffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:51.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.140+0000 7f3fcffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:51.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.140+0000 7f3fcffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:51.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.140+0000 7ffbf9046640 1 Processor -- start 2024-03-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.140+0000 7ffbf9046640 1 -- start start 2024-03-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.140+0000 7ffbf9046640 1 librados: setting wanted keys 2024-03-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.140+0000 7ffbf9046640 1 librados: calling monclient init 2024-03-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.140+0000 7ffbf9046640 10 monclient: init 2024-03-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.141+0000 7ffbf9046640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.141+0000 7ffbf9046640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.141+0000 7ffbf9046640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffbf4071480 0x7ffbf4133ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.141+0000 7ffbf9046640 10 monclient(hunting): picked mon.noname-c con 0x7ffbf4071480 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.141+0000 7ffbf9046640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbf40b32e0 0x7ffbf413c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.141+0000 7ffbf9046640 10 monclient(hunting): picked mon.noname-a con 0x7ffbf40b32e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.141+0000 7ffbf9046640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffbf413ee50 0x7ffbf4142810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.141+0000 7ffbf9046640 10 monclient(hunting): picked mon.noname-b con 0x7ffbf413ee50 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.141+0000 7ffbf9046640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.141+0000 7ffbf9046640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf4143d00 con 0x7ffbf40b32e0 2024-03-28T16:15:51.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.141+0000 7ffbf9046640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.141+0000 7ffbf9046640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf413dcc0 con 0x7ffbf413ee50 2024-03-28T16:15:51.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.141+0000 7ffbf9046640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.141+0000 7ffbf9046640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbf4134bf0 con 0x7ffbf4071480 2024-03-28T16:15:51.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.141+0000 7ffbf9046640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:51.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.141+0000 7ffbf9046640 10 monclient(hunting): authenticate will time out at 1418.283205s 2024-03-28T16:15:51.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.142+0000 7ffbf259c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbf40b32e0 0x7ffbf413c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.144+0000 7f3fcffff640 10 monclient(hunting): handle_auth_done global_id 4332 payload 824 2024-03-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.144+0000 7f3fcffff640 1 -- 172.21.15.7:0/3691783127 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3fd00a86e0 msgr2=0x7f3fd01d70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.144+0000 7f3fcffff640 1 --2- 172.21.15.7:0/3691783127 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3fd00a86e0 0x7f3fd01d70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.144+0000 7f3fcffff640 1 -- 172.21.15.7:0/3691783127 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fd0071480 msgr2=0x7f3fd01d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.144+0000 7f3fcffff640 1 --2- 172.21.15.7:0/3691783127 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fd0071480 0x7f3fd01d6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.144+0000 7f3fcffff640 10 monclient: _finish_hunting 0 2024-03-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.144+0000 7f3fcffff640 1 monclient: found mon.noname-a 2024-03-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.144+0000 7f3fcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.144+0000 7f3fcffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:15:51.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.144+0000 7f3fcffff640 1 -- 172.21.15.7:0/3691783127 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc8009b30 con 0x7f3fd00b32e0 2024-03-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.146+0000 7ffbf259c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbf40b32e0 0x7ffbf413c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38018/0 (socket says 172.21.15.7:38018) 2024-03-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.146+0000 7ffbf259c640 1 -- 172.21.15.7:0/3396548066 learned_addr learned my addr 172.21.15.7:0/3396548066 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.146+0000 7ffbf259c640 10 monclient(hunting): get_auth_request con 0x7ffbf40b32e0 auth_method 0 2024-03-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.146+0000 7ffbf259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.146+0000 7ffbf259c640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.146+0000 7ffbf259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.147+0000 7ffbf259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.147+0000 7ffbf259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.147+0000 7ffbf259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.148+0000 7f3fcffff640 1 --2- 172.21.15.7:0/3691783127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fd00b32e0 0x7f3fd0138380 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8036040 tx=0x7f3fc8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.148+0000 7f3fb7fff640 10 client.?.objecter ms_handle_connect 0x7f3fd00b32e0 2024-03-28T16:15:51.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.148+0000 7f3fb7fff640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:51.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.151+0000 7ffbf259c640 10 monclient(hunting): handle_auth_done global_id 4335 payload 824 2024-03-28T16:15:51.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.151+0000 7ffbf259c640 1 -- 172.21.15.7:0/3396548066 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffbf4071480 msgr2=0x7ffbf4133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.151+0000 7ffbf259c640 1 --2- 172.21.15.7:0/3396548066 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffbf4071480 0x7ffbf4133ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.151+0000 7ffbf259c640 1 -- 172.21.15.7:0/3396548066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffbf413ee50 msgr2=0x7ffbf4142810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.151+0000 7ffbf259c640 1 --2- 172.21.15.7:0/3396548066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffbf413ee50 0x7ffbf4142810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.151+0000 7ffbf259c640 10 monclient: _finish_hunting 0 2024-03-28T16:15:51.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.151+0000 7ffbf259c640 1 monclient: found mon.noname-a 2024-03-28T16:15:51.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.151+0000 7ffbf259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:51.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.151+0000 7ffbf259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:15:51.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.151+0000 7ffbf259c640 1 -- 172.21.15.7:0/3396548066 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbec009b30 con 0x7ffbf40b32e0 2024-03-28T16:15:51.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7ffbdb7fe640 10 client.?.objecter ms_handle_connect 0x7ffbf40b32e0 2024-03-28T16:15:51.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7ffbdb7fe640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:51.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7ffbf259c640 1 --2- 172.21.15.7:0/3396548066 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbf40b32e0 0x7ffbf413c3a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ffbe800c400 tx=0x7ffbe800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fb7fff640 1 -- 172.21.15.7:0/3691783127 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc803e450 con 0x7f3fd00b32e0 2024-03-28T16:15:51.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fb7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:15:51.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fb7fff640 10 monclient: dump: 2024-03-28T16:15:51.204 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.204 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.204 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.204 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.204 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.204 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.205 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fb7fff640 10 monclient: _finish_auth 0 2024-03-28T16:15:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fb7fff640 10 monclient: _check_auth_tickets 2024-03-28T16:15:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fb7fff640 1 -- 172.21.15.7:0/3691783127 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc803e5d0 con 0x7f3fd00b32e0 2024-03-28T16:15:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fb7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fb7fff640 1 -- 172.21.15.7:0/3691783127 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc803e980 con 0x7f3fd00b32e0 2024-03-28T16:15:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fb7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:15:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fb7fff640 10 monclient: dump: 2024-03-28T16:15:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.206 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fd5f37640 5 monclient: authenticate success, global_id 4332 2024-03-28T16:15:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fd5f37640 10 monclient: _renew_subs 2024-03-28T16:15:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fd5f37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fd5f37640 1 -- 172.21.15.7:0/3691783127 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fd013d420 con 0x7f3fd00b32e0 2024-03-28T16:15:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fcd7fa640 4 set_mon_vals no callback set 2024-03-28T16:15:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fd5f37640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:51.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fd5f37640 10 monclient: _renew_subs 2024-03-28T16:15:51.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fd5f37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:51.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fd5f37640 1 -- 172.21.15.7:0/3691783127 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fd0144810 con 0x7f3fd00b32e0 2024-03-28T16:15:51.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.157+0000 7f3fd5f37640 1 librados: init done 2024-03-28T16:15:51.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.158+0000 7f3fd5f37640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.159+0000 7f3fd5f37640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.159+0000 7f3fd5f37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:51.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.159+0000 7f3fd5f37640 1 -- 172.21.15.7:0/3691783127 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd01d6810 con 0x7f3fd00b32e0 2024-03-28T16:15:51.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.167+0000 7ffbdb7fe640 1 -- 172.21.15.7:0/3396548066 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe80052d0 con 0x7ffbf40b32e0 2024-03-28T16:15:51.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.167+0000 7ffbdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.167+0000 7ffbdb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:15:51.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.167+0000 7ffbdb7fe640 10 monclient: dump: 2024-03-28T16:15:51.208 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.208 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.208 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.208 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.209 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.167+0000 7ffbdb7fe640 10 monclient: _finish_auth 0 2024-03-28T16:15:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.167+0000 7ffbdb7fe640 10 monclient: _check_auth_tickets 2024-03-28T16:15:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.167+0000 7ffbdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.167+0000 7ffbf9046640 5 monclient: authenticate success, global_id 4335 2024-03-28T16:15:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.167+0000 7ffbf9046640 10 monclient: _renew_subs 2024-03-28T16:15:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.167+0000 7ffbf9046640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.167+0000 7ffbf9046640 1 -- 172.21.15.7:0/3396548066 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbf4142d50 con 0x7ffbf40b32e0 2024-03-28T16:15:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.167+0000 7ffbf9046640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.167+0000 7ffbf9046640 10 monclient: _renew_subs 2024-03-28T16:15:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.167+0000 7ffbf9046640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.167+0000 7ffbf9046640 1 -- 172.21.15.7:0/3396548066 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbf413d980 con 0x7ffbf40b32e0 2024-03-28T16:15:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.167+0000 7ffbf9046640 1 librados: init done 2024-03-28T16:15:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.169+0000 7ffbf9046640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.169+0000 7ffbf9046640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.169+0000 7ffbf9046640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.169+0000 7ffbf9046640 1 -- 172.21.15.7:0/3396548066 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbf40aa370 con 0x7ffbf40b32e0 2024-03-28T16:15:51.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.177+0000 7faafebce640 10 monclient: get_monmap_and_config 2024-03-28T16:15:51.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.177+0000 7faafebce640 10 monclient: build_initial_monmap 2024-03-28T16:15:51.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.177+0000 7faafebce640 10 monclient: monmap: 2024-03-28T16:15:51.212 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:51.212 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.212 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:51.178992+0000 2024-03-28T16:15:51.212 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:51.178992+0000 2024-03-28T16:15:51.212 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:51.212 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.212 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:51.212 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:51.212 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:51.212 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:51.212 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:51.212 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.178+0000 7faafebce640 1 Processor -- start 2024-03-28T16:15:51.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.178+0000 7faafebce640 1 -- start start 2024-03-28T16:15:51.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.178+0000 7faafebce640 10 monclient: init 2024-03-28T16:15:51.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.178+0000 7faafebce640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:51.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.178+0000 7faafebce640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-03-28T16:15:51.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.178+0000 7faafebce640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faaf8071920 addr v1:172.21.15.83:6789/0 2024-03-28T16:15:51.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.178+0000 7faafebce640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faaf80a89a0 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:51.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.178+0000 7faafebce640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7faaf813ef40 0x7faaf80b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.178+0000 7faafebce640 10 monclient(hunting): picked mon.noname-c con 0x7faaf813ef40 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:51.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.178+0000 7faafebce640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf80a8ec0 con 0x7faaf80a89a0 2024-03-28T16:15:51.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.178+0000 7faafebce640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf80aa660 con 0x7faaf8071920 2024-03-28T16:15:51.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.178+0000 7faafebce640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.178+0000 7faafebce640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaf80aa480 con 0x7faaf813ef40 2024-03-28T16:15:51.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.178+0000 7faafebce640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:51.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.178+0000 7faafebce640 10 monclient(hunting): authenticate will time out at 1418.320823s 2024-03-28T16:15:51.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.211+0000 7faaf7fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7faaf80a89a0 0x7faaf80b1890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54490/0 (socket says 172.21.15.7:54490) 2024-03-28T16:15:51.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.211+0000 7faaf7fff640 1 -- 172.21.15.7:0/3317459426 learned_addr learned my addr 172.21.15.7:0/3317459426 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-28T16:15:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.386+0000 7f95b17fa640 1 -- 172.21.15.7:0/3321025491 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f95a0003ca0 con 0x7f95ac035640 2024-03-28T16:15:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.386+0000 7faafd16b640 1 --2- 172.21.15.7:0/3317459426 >> v2:172.21.15.83:3300/0 conn(0x7faaf813ef40 0x7faaf80b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.386+0000 7faafd16b640 10 monclient(hunting): get_auth_request con 0x7faaf813ef40 auth_method 0 2024-03-28T16:15:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.386+0000 7faafd16b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.386+0000 7faafd16b640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.386+0000 7faafd16b640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.386+0000 7f95b17fa640 1 --2- 172.21.15.7:0/3321025491 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f959806c590 0x7f959806ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.386+0000 7f95b3fff640 1 --2- 172.21.15.7:0/3321025491 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f959806c590 0x7f959806ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.386+0000 7f95b3fff640 10 monclient: get_auth_request con 0x7f959806c590 auth_method 0 2024-03-28T16:15:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.387+0000 7f95b3fff640 1 --2- 172.21.15.7:0/3321025491 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f959806c590 0x7f959806ea50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f95a8036000 tx=0x7f95a803c040 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.387+0000 7f95b17fa640 10 client.4363.objecter ms_handle_connect 0x7f959806c590 2024-03-28T16:15:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.390+0000 7f95b17fa640 1 -- 172.21.15.7:0/3321025491 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f95a0057f40 con 0x7f95ac035640 2024-03-28T16:15:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.390+0000 7f95b17fa640 10 client.4363.objecter ms_dispatch 0x7f95b4000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:51.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.390+0000 7f95b17fa640 3 client.4363.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.390+0000 7f95b17fa640 3 client.4363.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:51.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.390+0000 7f95b17fa640 20 client.4363.objecter dump_active .. 0 homeless 2024-03-28T16:15:51.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.397+0000 7f1afe7fc640 1 -- 172.21.15.7:0/1829584434 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0031740 con 0x7f1b00071480 2024-03-28T16:15:51.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.397+0000 7f1afe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.397+0000 7f1afe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:51.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.397+0000 7f1afe7fc640 10 monclient: dump: 2024-03-28T16:15:51.399 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.399 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.399 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.399 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.399 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.399 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.399 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.399 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.400 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.400 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.397+0000 7f1afe7fc640 10 monclient: _finish_auth 0 2024-03-28T16:15:51.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.397+0000 7f1afe7fc640 10 monclient: _check_auth_tickets 2024-03-28T16:15:51.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.397+0000 7f1afe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:51.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.397+0000 7f1afe7fc640 1 -- 172.21.15.7:0/1829584434 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af00318a0 con 0x7f1b00071480 2024-03-28T16:15:51.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.397+0000 7f1b05b9c640 5 monclient: authenticate success, global_id 4295 2024-03-28T16:15:51.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.397+0000 7f1afe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:51.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.397+0000 7f1b05b9c640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:51.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.397+0000 7f1afe7fc640 1 -- 172.21.15.7:0/1829584434 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0031c30 con 0x7f1b00071480 2024-03-28T16:15:51.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.397+0000 7f1b05b9c640 4 set_mon_vals no callback set 2024-03-28T16:15:51.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.397+0000 7f1afe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.397+0000 7f1afe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:15:51.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.397+0000 7f1afe7fc640 10 monclient: dump: 2024-03-28T16:15:51.401 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.401 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.401 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.401 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.401 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.401 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.401 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.401 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.401 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.401 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.398+0000 7f1b05b9c640 10 monclient: shutdown 2024-03-28T16:15:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.398+0000 7f1b05b9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:51.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.398+0000 7f1b05b9c640 1 -- 172.21.15.7:0/1829584434 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1b00071480 msgr2=0x7f1b0013e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.398+0000 7f1b05b9c640 1 --2- 172.21.15.7:0/1829584434 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1b00071480 0x7f1b0013e910 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1af0009870 tx=0x7f1af002b7e0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.398+0000 7f1b05b9c640 1 -- 172.21.15.7:0/1829584434 shutdown_connections 2024-03-28T16:15:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.398+0000 7f1b05b9c640 1 --2- 172.21.15.7:0/1829584434 >> v2:172.21.15.83:3300/0 conn(0x7f1b0013ee50 0x7f1b00145110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.398+0000 7f1b05b9c640 1 --2- 172.21.15.7:0/1829584434 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1b00071480 0x7f1b0013e910 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.398+0000 7f1b05b9c640 1 -- 172.21.15.7:0/1829584434 >> 172.21.15.7:0/1829584434 conn(0x7f1b000b70e0 msgr2=0x7f1b000b74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.398+0000 7f1b05b9c640 1 -- 172.21.15.7:0/1829584434 shutdown_connections 2024-03-28T16:15:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.398+0000 7f1b05b9c640 1 -- 172.21.15.7:0/1829584434 wait complete. 2024-03-28T16:15:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.399+0000 7f1b05b9c640 10 monclient: build_initial_monmap 2024-03-28T16:15:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.399+0000 7f1b05b9c640 10 monclient: monmap: 2024-03-28T16:15:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:51.404 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:51.405 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.399+0000 7f1b05b9c640 1 librados: starting msgr at 2024-03-28T16:15:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.399+0000 7f1b05b9c640 1 librados: starting objecter 2024-03-28T16:15:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.399+0000 7f1b05b9c640 1 Processor -- start 2024-03-28T16:15:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.399+0000 7f1b05b9c640 1 -- start start 2024-03-28T16:15:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.399+0000 7f1b05b9c640 1 librados: setting wanted keys 2024-03-28T16:15:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.399+0000 7f1b05b9c640 1 librados: calling monclient init 2024-03-28T16:15:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.399+0000 7f1b05b9c640 10 monclient: init 2024-03-28T16:15:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.400+0000 7f1b05b9c640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.400+0000 7f1b05b9c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-28T16:15:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.400+0000 7f1b05b9c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1b00071480 0x7f1b0013de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.400+0000 7f1b05b9c640 10 monclient(hunting): picked mon.noname-c con 0x7f1b00071480 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.400+0000 7f1b05b9c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b000b32e0 0x7f1b00143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.400+0000 7f1b05b9c640 10 monclient(hunting): picked mon.noname-a con 0x7f1b000b32e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.400+0000 7f1b05b9c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b0013ee50 0x7f1b00144020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.400+0000 7f1b05b9c640 10 monclient(hunting): picked mon.noname-b con 0x7f1b0013ee50 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.400+0000 7f1b05b9c640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.400+0000 7f1b05b9c640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0013da00 con 0x7f1b000b32e0 2024-03-28T16:15:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.400+0000 7f1b05b9c640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.400+0000 7f1b05b9c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b001351c0 con 0x7f1b0013ee50 2024-03-28T16:15:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.400+0000 7f1b05b9c640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.400+0000 7f1b05b9c640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b001348d0 con 0x7f1b00071480 2024-03-28T16:15:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.400+0000 7f1b05b9c640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:51.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.400+0000 7f1b05b9c640 10 monclient(hunting): authenticate will time out at 1418.542327s 2024-03-28T16:15:51.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.410+0000 7fe794ff9640 1 -- 172.21.15.7:0/3116337946 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe78404d3f0 con 0x7fe790032ec0 2024-03-28T16:15:51.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.411+0000 7fe794ff9640 1 --2- 172.21.15.7:0/3116337946 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fe77c06c5e0 0x7fe77c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.411+0000 7fe797fff640 1 --2- 172.21.15.7:0/3116337946 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fe77c06c5e0 0x7fe77c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.411+0000 7fe797fff640 10 monclient: get_auth_request con 0x7fe77c06c5e0 auth_method 0 2024-03-28T16:15:51.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.411+0000 7fe794ff9640 10 client.4292.objecter ms_handle_connect 0x7fe77c06c5e0 2024-03-28T16:15:51.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.411+0000 7fe797fff640 1 --2- 172.21.15.7:0/3116337946 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fe77c06c5e0 0x7fe77c06eaa0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe78c009890 tx=0x7fe78c008040 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.616+0000 7fe794ff9640 1 -- 172.21.15.7:0/3116337946 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe78407a980 con 0x7fe790032ec0 2024-03-28T16:15:51.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.616+0000 7fe794ff9640 10 client.4292.objecter ms_dispatch 0x7fe798000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:51.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.616+0000 7fe794ff9640 3 client.4292.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:51.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.616+0000 7fe794ff9640 3 client.4292.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:51.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.616+0000 7fe794ff9640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-28T16:15:51.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.629+0000 7faafd16b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:51.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.629+0000 7faafd16b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:51.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.629+0000 7faafd16b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:51.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.631+0000 7faafd16b640 10 monclient(hunting): handle_auth_done global_id 4372 payload 274 2024-03-28T16:15:51.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.631+0000 7faafd16b640 1 -- 172.21.15.7:0/3317459426 >> v1:172.21.15.83:6789/0 conn(0x7faaf8071920 legacy=0x7faaf813e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:51.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.631+0000 7faafd16b640 1 -- 172.21.15.7:0/3317459426 >> v1:172.21.15.7:6789/0 conn(0x7faaf80a89a0 legacy=0x7faaf80b1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.631+0000 7faafd16b640 10 monclient: _finish_hunting 0 2024-03-28T16:15:51.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.631+0000 7faafd16b640 1 monclient: found mon.noname-c 2024-03-28T16:15:51.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.631+0000 7faafd16b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:51.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.631+0000 7faafd16b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3300/0 2024-03-28T16:15:51.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.631+0000 7faafd16b640 1 -- 172.21.15.7:0/3317459426 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf80b36d0 con 0x7faaf813ef40 2024-03-28T16:15:51.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.649+0000 7f9d567fc640 1 -- 172.21.15.7:0/2093165412 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9d580052e0 con 0x7f9d6013ee50 2024-03-28T16:15:51.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.649+0000 7f9d567fc640 1 --2- 172.21.15.7:0/2093165412 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f9d4406c5e0 0x7f9d4406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.649+0000 7f9d65e4b640 1 --2- 172.21.15.7:0/2093165412 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f9d4406c5e0 0x7f9d4406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.649+0000 7f9d65e4b640 10 monclient: get_auth_request con 0x7f9d4406c5e0 auth_method 0 2024-03-28T16:15:51.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.650+0000 7f9d65e4b640 1 --2- 172.21.15.7:0/2093165412 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f9d4406c5e0 0x7f9d4406eaa0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9d50009710 tx=0x7f9d50006c90 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.650+0000 7f9d567fc640 10 client.4360.objecter ms_handle_connect 0x7f9d4406c5e0 2024-03-28T16:15:51.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.653+0000 7f9d567fc640 1 -- 172.21.15.7:0/2093165412 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9d580593f0 con 0x7f9d6013ee50 2024-03-28T16:15:51.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.653+0000 7f9d567fc640 10 client.4360.objecter ms_dispatch 0x7f9d60000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:51.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.653+0000 7f9d567fc640 3 client.4360.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:51.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.653+0000 7f9d567fc640 3 client.4360.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:51.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.653+0000 7f9d567fc640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-28T16:15:51.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.665+0000 7f9d567fc640 1 -- 172.21.15.7:0/2093165412 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9d58055030 con 0x7f9d6013ee50 2024-03-28T16:15:51.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.666+0000 7f9d567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.666+0000 7f9d567fc640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:51.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.666+0000 7f95b17fa640 1 -- 172.21.15.7:0/3321025491 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f95a0053b80 con 0x7f95ac035640 2024-03-28T16:15:51.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.666+0000 7f95b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.666+0000 7f95b17fa640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:51.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.668+0000 7faafd16b640 1 --2- 172.21.15.7:0/3317459426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faaf813ef40 0x7faaf80b1fa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7faaf0009d90 tx=0x7faaf002be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6540bee2e666c513 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.673+0000 7faaf77fe640 1 -- 172.21.15.7:0/3317459426 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf002cbd0 con 0x7faaf813ef40 2024-03-28T16:15:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.673+0000 7faaf77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.673+0000 7faaf77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.673+0000 7faaf77fe640 10 monclient: dump: 2024-03-28T16:15:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.675 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.673+0000 7faaf77fe640 10 monclient: _finish_auth 0 2024-03-28T16:15:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.673+0000 7faaf77fe640 10 monclient: _check_auth_tickets 2024-03-28T16:15:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.673+0000 7faaf77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.673+0000 7faaf77fe640 1 -- 172.21.15.7:0/3317459426 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf002cd30 con 0x7faaf813ef40 2024-03-28T16:15:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.673+0000 7faafebce640 5 monclient: authenticate success, global_id 4372 2024-03-28T16:15:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.673+0000 7faafebce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.673+0000 7faaf77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.673+0000 7faafebce640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.673+0000 7faafebce640 4 set_mon_vals no callback set 2024-03-28T16:15:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.673+0000 7faaf77fe640 1 -- 172.21.15.7:0/3317459426 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf002cea0 con 0x7faaf813ef40 2024-03-28T16:15:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.673+0000 7faaf77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.673+0000 7faaf77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.673+0000 7faaf77fe640 10 monclient: dump: 2024-03-28T16:15:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.677 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.677 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.677 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.677 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.677 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.677 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.677 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.674+0000 7f1afffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b0013ee50 0x7f1b00144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.674+0000 7faafebce640 10 monclient: shutdown 2024-03-28T16:15:51.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.674+0000 7faafebce640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:51.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.674+0000 7faafebce640 1 -- 172.21.15.7:0/3317459426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faaf813ef40 msgr2=0x7faaf80b1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.674+0000 7faafebce640 1 --2- 172.21.15.7:0/3317459426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faaf813ef40 0x7faaf80b1fa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7faaf0009d90 tx=0x7faaf002be20 comp rx=0 tx=0).stop 2024-03-28T16:15:51.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.674+0000 7faafebce640 1 -- 172.21.15.7:0/3317459426 shutdown_connections 2024-03-28T16:15:51.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.674+0000 7faafebce640 1 --2- 172.21.15.7:0/3317459426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faaf813ef40 0x7faaf80b1fa0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.674+0000 7faafebce640 1 -- 172.21.15.7:0/3317459426 >> 172.21.15.7:0/3317459426 conn(0x7faaf80b3db0 msgr2=0x7faaf80b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:51.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.674+0000 7faafebce640 1 -- 172.21.15.7:0/3317459426 shutdown_connections 2024-03-28T16:15:51.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.674+0000 7faafebce640 1 -- 172.21.15.7:0/3317459426 wait complete. 2024-03-28T16:15:51.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.675+0000 7f1afffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b0013ee50 0x7f1b00144020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:58214/0 (socket says 172.21.15.7:58214) 2024-03-28T16:15:51.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.675+0000 7f1afffff640 1 -- 172.21.15.7:0/337397591 learned_addr learned my addr 172.21.15.7:0/337397591 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:51.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.675+0000 7f1afffff640 10 monclient(hunting): get_auth_request con 0x7f1b0013ee50 auth_method 0 2024-03-28T16:15:51.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.675+0000 7f1afffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:51.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.675+0000 7f1afffff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:51.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.675+0000 7f1afffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:51.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.675+0000 7faafebce640 10 monclient: build_initial_monmap 2024-03-28T16:15:51.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.675+0000 7faafebce640 10 monclient: monmap: 2024-03-28T16:15:51.680 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:51.681 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:51.681 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:51.681 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:51.681 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:51.681 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.681 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:51.681 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:51.681 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:51.681 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.675+0000 7faafebce640 1 librados: starting msgr at 2024-03-28T16:15:51.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.675+0000 7faafebce640 1 librados: starting objecter 2024-03-28T16:15:51.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.675+0000 7faafebce640 1 Processor -- start 2024-03-28T16:15:51.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7faafebce640 1 -- start start 2024-03-28T16:15:51.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7faafebce640 1 librados: setting wanted keys 2024-03-28T16:15:51.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7faafebce640 1 librados: calling monclient init 2024-03-28T16:15:51.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7faafebce640 10 monclient: init 2024-03-28T16:15:51.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7faafebce640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:51.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7faafebce640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-28T16:15:51.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7f1afffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:51.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7f1afffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:51.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7f1afffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:51.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7faafebce640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faaf8071920 0x7faaf81442f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7faafebce640 10 monclient(hunting): picked mon.noname-c con 0x7faaf8071920 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:51.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7faafebce640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faaf813ef40 0x7faaf8141680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7faafebce640 10 monclient(hunting): picked mon.noname-a con 0x7faaf813ef40 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:51.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7faafebce640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faaf8142850 0x7faaf8144f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7faafebce640 10 monclient(hunting): picked mon.noname-b con 0x7faaf8142850 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:51.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7faafebce640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7faafebce640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf8136f10 con 0x7faaf813ef40 2024-03-28T16:15:51.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7faafebce640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7faafebce640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf81370f0 con 0x7faaf8142850 2024-03-28T16:15:51.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7faafebce640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:51.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7faafebce640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf81373d0 con 0x7faaf8071920 2024-03-28T16:15:51.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7faafebce640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:51.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.676+0000 7faafebce640 10 monclient(hunting): authenticate will time out at 1418.818734s 2024-03-28T16:15:51.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.679+0000 7f1afffff640 10 monclient(hunting): handle_auth_done global_id 4375 payload 824 2024-03-28T16:15:51.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.679+0000 7f1afffff640 1 -- 172.21.15.7:0/337397591 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1b00071480 msgr2=0x7f1b0013de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.679+0000 7f1afffff640 1 --2- 172.21.15.7:0/337397591 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1b00071480 0x7f1b0013de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.679+0000 7f1afffff640 1 -- 172.21.15.7:0/337397591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b000b32e0 msgr2=0x7f1b00143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.679+0000 7f1afffff640 1 --2- 172.21.15.7:0/337397591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b000b32e0 0x7f1b00143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.679+0000 7f1afffff640 10 monclient: _finish_hunting 0 2024-03-28T16:15:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.679+0000 7f1afffff640 1 monclient: found mon.noname-b 2024-03-28T16:15:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.679+0000 7f1afffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.679+0000 7f1afffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:15:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.679+0000 7f1afffff640 1 -- 172.21.15.7:0/337397591 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af00094d0 con 0x7f1b0013ee50 2024-03-28T16:15:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.680+0000 7faafd16b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faaf8142850 0x7faaf8144f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.681+0000 7fe794ff9640 1 -- 172.21.15.7:0/3116337946 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe7840765c0 con 0x7fe790032ec0 2024-03-28T16:15:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.681+0000 7fe794ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.681+0000 7fe794ff9640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.681+0000 7f1afffff640 1 --2- 172.21.15.7:0/337397591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b0013ee50 0x7f1b00144020 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1af8009cb0 tx=0x7f1af800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.681+0000 7f1ae7fff640 10 client.?.objecter ms_handle_connect 0x7f1b0013ee50 2024-03-28T16:15:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.681+0000 7f1ae7fff640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.682+0000 7faafd16b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faaf8142850 0x7faaf8144f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:58222/0 (socket says 172.21.15.7:58222) 2024-03-28T16:15:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.682+0000 7faafd16b640 1 -- 172.21.15.7:0/300693767 learned_addr learned my addr 172.21.15.7:0/300693767 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.682+0000 7faafd16b640 10 monclient(hunting): get_auth_request con 0x7faaf8142850 auth_method 0 2024-03-28T16:15:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.682+0000 7faafd16b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.682+0000 7faafd16b640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.682+0000 7faafd16b640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.684+0000 7faafd16b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.684+0000 7faafd16b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.684+0000 7faafd16b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.688+0000 7faafd16b640 10 monclient(hunting): handle_auth_done global_id 4378 payload 824 2024-03-28T16:15:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.688+0000 7faafd16b640 1 -- 172.21.15.7:0/300693767 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faaf8071920 msgr2=0x7faaf81442f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.688+0000 7faafd16b640 1 --2- 172.21.15.7:0/300693767 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faaf8071920 0x7faaf81442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.688+0000 7faafd16b640 1 -- 172.21.15.7:0/300693767 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faaf813ef40 msgr2=0x7faaf8141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.688+0000 7faafd16b640 1 --2- 172.21.15.7:0/300693767 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faaf813ef40 0x7faaf8141680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.688+0000 7faafd16b640 10 monclient: _finish_hunting 0 2024-03-28T16:15:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.688+0000 7faafd16b640 1 monclient: found mon.noname-b 2024-03-28T16:15:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.688+0000 7faafd16b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:51.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.688+0000 7faafd16b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:15:51.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.688+0000 7faafd16b640 1 -- 172.21.15.7:0/300693767 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf0009a70 con 0x7faaf8142850 2024-03-28T16:15:51.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.688+0000 7f3fb7fff640 1 -- 172.21.15.7:0/3691783127 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3fc803eae0 con 0x7f3fd00b32e0 2024-03-28T16:15:51.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.688+0000 7f3fb7fff640 1 --2- 172.21.15.7:0/3691783127 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f3fbc06c610 0x7f3fbc06ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.688+0000 7f3fcf7fe640 1 --2- 172.21.15.7:0/3691783127 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f3fbc06c610 0x7f3fbc06ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.689+0000 7f3fcf7fe640 10 monclient: get_auth_request con 0x7f3fbc06c610 auth_method 0 2024-03-28T16:15:51.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.689+0000 7f3fcf7fe640 1 --2- 172.21.15.7:0/3691783127 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f3fbc06c610 0x7f3fbc06ead0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0001560 tx=0x7f3fc0007990 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.689+0000 7f3fb7fff640 10 client.4332.objecter ms_handle_connect 0x7f3fbc06c610 2024-03-28T16:15:51.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1ae7fff640 1 -- 172.21.15.7:0/337397591 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af8009b20 con 0x7f1b0013ee50 2024-03-28T16:15:51.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1ae7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1ae7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:51.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1ae7fff640 10 monclient: dump: 2024-03-28T16:15:51.692 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.692 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.692 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.693 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.693 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.693 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.693 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.693 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.693 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.693 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1ae7fff640 10 monclient: _finish_auth 0 2024-03-28T16:15:51.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1ae7fff640 10 monclient: _check_auth_tickets 2024-03-28T16:15:51.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1ae7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:51.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1ae7fff640 1 -- 172.21.15.7:0/337397591 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af800f040 con 0x7f1b0013ee50 2024-03-28T16:15:51.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1ae7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:51.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1ae7fff640 1 -- 172.21.15.7:0/337397591 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af80146c0 con 0x7f1b0013ee50 2024-03-28T16:15:51.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1ae7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1ae7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1ae7fff640 10 monclient: dump: 2024-03-28T16:15:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.694 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1b05b9c640 5 monclient: authenticate success, global_id 4375 2024-03-28T16:15:51.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1b05b9c640 10 monclient: _renew_subs 2024-03-28T16:15:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1afd7fa640 4 set_mon_vals no callback set 2024-03-28T16:15:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1b05b9c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1b05b9c640 1 -- 172.21.15.7:0/337397591 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b00144560 con 0x7f1b0013ee50 2024-03-28T16:15:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1b05b9c640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1b05b9c640 10 monclient: _renew_subs 2024-03-28T16:15:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1b05b9c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1b05b9c640 1 -- 172.21.15.7:0/337397591 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b0013cfa0 con 0x7f1b0013ee50 2024-03-28T16:15:51.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.691+0000 7f1b05b9c640 1 librados: init done 2024-03-28T16:15:51.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.692+0000 7f1b05b9c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.692+0000 7f1b05b9c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.692+0000 7f1b05b9c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:51.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.692+0000 7f1b05b9c640 1 -- 172.21.15.7:0/337397591 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b00144780 con 0x7f1b0013ee50 2024-03-28T16:15:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.704+0000 7f1696ffd640 1 -- 172.21.15.7:0/2557141569 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1690023480 con 0x7f16a00b32e0 2024-03-28T16:15:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.705+0000 7f1696ffd640 1 --2- 172.21.15.7:0/2557141569 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f168406c5e0 0x7f168406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.705+0000 7f16a6277640 1 --2- 172.21.15.7:0/2557141569 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f168406c5e0 0x7f168406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.705+0000 7f16a6277640 10 monclient: get_auth_request con 0x7f168406c5e0 auth_method 0 2024-03-28T16:15:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.705+0000 7f1696ffd640 10 client.4286.objecter ms_handle_connect 0x7f168406c5e0 2024-03-28T16:15:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.705+0000 7f16a6277640 1 --2- 172.21.15.7:0/2557141569 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f168406c5e0 0x7f168406eaa0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f169c0094a0 tx=0x7f169c0093f0 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.710+0000 7f1ae7fff640 1 -- 172.21.15.7:0/337397591 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1af8014820 con 0x7f1b0013ee50 2024-03-28T16:15:51.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.710+0000 7f1ae7fff640 1 --2- 172.21.15.7:0/337397591 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f1aec06c590 0x7f1aec06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.711+0000 7f1aff7fe640 1 --2- 172.21.15.7:0/337397591 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f1aec06c590 0x7f1aec06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.711+0000 7f1aff7fe640 10 monclient: get_auth_request con 0x7f1aec06c590 auth_method 0 2024-03-28T16:15:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.711+0000 7f1aff7fe640 1 --2- 172.21.15.7:0/337397591 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f1aec06c590 0x7f1aec06ea50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1af0035000 tx=0x7f1af0030000 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.712+0000 7f1ae7fff640 10 client.4375.objecter ms_handle_connect 0x7f1aec06c590 2024-03-28T16:15:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.712+0000 7faafd16b640 1 --2- 172.21.15.7:0/300693767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faaf8142850 0x7faaf8144f00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faaf0009eb0 tx=0x7faaf0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.712+0000 7faaf57fa640 10 client.?.objecter ms_handle_connect 0x7faaf8142850 2024-03-28T16:15:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.712+0000 7faaf57fa640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:51.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.713+0000 7f1ae7fff640 1 -- 172.21.15.7:0/337397591 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1af8059b10 con 0x7f1b0013ee50 2024-03-28T16:15:51.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.713+0000 7f1ae7fff640 10 client.4375.objecter ms_dispatch 0x7f1b00000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:51.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.713+0000 7f1ae7fff640 3 client.4375.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:51.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.713+0000 7f1ae7fff640 3 client.4375.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:51.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.713+0000 7f1ae7fff640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-28T16:15:51.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.717+0000 7ffbdb7fe640 1 -- 172.21.15.7:0/3396548066 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe8005430 con 0x7ffbf40b32e0 2024-03-28T16:15:51.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.717+0000 7ffbdb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:51.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.717+0000 7ffbdb7fe640 1 -- 172.21.15.7:0/3396548066 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe80052d0 con 0x7ffbf40b32e0 2024-03-28T16:15:51.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.717+0000 7ffbdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.717+0000 7ffbdb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:15:51.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.717+0000 7ffbdb7fe640 10 monclient: dump: 2024-03-28T16:15:51.720 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.720 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.720 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.720 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.720 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.720 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.720 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.720 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.720 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.720 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.717+0000 7ffbf159a640 4 set_mon_vals no callback set 2024-03-28T16:15:51.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.720+0000 7ffbdb7fe640 1 -- 172.21.15.7:0/3396548066 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ffbe8007020 con 0x7ffbf40b32e0 2024-03-28T16:15:51.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.720+0000 7ffbdb7fe640 1 --2- 172.21.15.7:0/3396548066 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7ffbe006c5f0 0x7ffbe006eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:51.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.720+0000 7ffbf2d9d640 1 --2- 172.21.15.7:0/3396548066 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7ffbe006c5f0 0x7ffbe006eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:51.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.720+0000 7ffbf2d9d640 10 monclient: get_auth_request con 0x7ffbe006c5f0 auth_method 0 2024-03-28T16:15:51.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.721+0000 7ffbf2d9d640 1 --2- 172.21.15.7:0/3396548066 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7ffbe006c5f0 0x7ffbe006eab0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ffbe40098f0 tx=0x7ffbe4008040 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:51.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.721+0000 7ffbdb7fe640 10 client.4335.objecter ms_handle_connect 0x7ffbe006c5f0 2024-03-28T16:15:51.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.729+0000 7f1696ffd640 1 -- 172.21.15.7:0/2557141569 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1690058d10 con 0x7f16a00b32e0 2024-03-28T16:15:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.729+0000 7f1696ffd640 10 client.4286.objecter ms_dispatch 0x7f16a0000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.729+0000 7f1696ffd640 3 client.4286.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.729+0000 7f1696ffd640 3 client.4286.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.730+0000 7f1696ffd640 20 client.4286.objecter dump_active .. 0 homeless 2024-03-28T16:15:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.730+0000 7f1696ffd640 1 -- 172.21.15.7:0/2557141569 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1690054950 con 0x7f16a00b32e0 2024-03-28T16:15:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.730+0000 7f1696ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.730+0000 7f1696ffd640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:51.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.736+0000 7f3fb7fff640 1 -- 172.21.15.7:0/3691783127 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3fc807be10 con 0x7f3fd00b32e0 2024-03-28T16:15:51.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.736+0000 7f3fb7fff640 10 client.4332.objecter ms_dispatch 0x7f3fd0000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:51.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.736+0000 7f3fb7fff640 3 client.4332.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:51.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.736+0000 7f3fb7fff640 3 client.4332.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:51.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.736+0000 7f3fb7fff640 20 client.4332.objecter dump_active .. 0 homeless 2024-03-28T16:15:51.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.736+0000 7f3fb7fff640 1 -- 172.21.15.7:0/3691783127 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3fc8077a50 con 0x7f3fd00b32e0 2024-03-28T16:15:51.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.736+0000 7f3fb7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.736+0000 7f3fb7fff640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:51.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.753+0000 7ffbdb7fe640 1 -- 172.21.15.7:0/3396548066 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ffbe8057c00 con 0x7ffbf40b32e0 2024-03-28T16:15:51.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.753+0000 7ffbdb7fe640 10 client.4335.objecter ms_dispatch 0x7ffbf4000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:51.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.753+0000 7ffbdb7fe640 3 client.4335.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:51.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.753+0000 7ffbdb7fe640 3 client.4335.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:51.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.753+0000 7ffbdb7fe640 20 client.4335.objecter dump_active .. 0 homeless 2024-03-28T16:15:51.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.753+0000 7ffbdb7fe640 1 -- 172.21.15.7:0/3396548066 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffbe8053fd0 con 0x7ffbf40b32e0 2024-03-28T16:15:51.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.753+0000 7ffbdb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.753+0000 7ffbdb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:51.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.939+0000 7f16a84db640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-28T16:15:51.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.939+0000 7f16a84db640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-28T16:15:51.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.939+0000 7f16a84db640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:51.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.939+0000 7f16a84db640 1 -- 172.21.15.7:0/2557141569 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f16a013a480 con 0x7f16a00b32e0 2024-03-28T16:15:51.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faaf57fa640 1 -- 172.21.15.7:0/300693767 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf002cea0 con 0x7faaf8142850 2024-03-28T16:15:51.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faaf57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faaf57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:51.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faaf57fa640 10 monclient: dump: 2024-03-28T16:15:51.942 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.942 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.943 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.943 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.943 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.943 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.943 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.943 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.943 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.943 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faaf57fa640 10 monclient: _finish_auth 0 2024-03-28T16:15:51.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faaf57fa640 10 monclient: _check_auth_tickets 2024-03-28T16:15:51.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faaf57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:51.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faaf57fa640 1 -- 172.21.15.7:0/300693767 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf003f420 con 0x7faaf8142850 2024-03-28T16:15:51.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faaf57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:51.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faaf57fa640 1 -- 172.21.15.7:0/300693767 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf003f7b0 con 0x7faaf8142850 2024-03-28T16:15:51.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faaf57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:51.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faaf57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:51.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faaf57fa640 10 monclient: dump: 2024-03-28T16:15:51.944 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:51.944 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:51.944 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.944 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:51.944 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:51.944 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:51.944 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:51.944 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:51.945 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:51.945 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:51.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faafebce640 5 monclient: authenticate success, global_id 4378 2024-03-28T16:15:51.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faafebce640 10 monclient: _renew_subs 2024-03-28T16:15:51.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faaf6ffd640 4 set_mon_vals no callback set 2024-03-28T16:15:51.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faafebce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:51.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faafebce640 1 -- 172.21.15.7:0/300693767 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf81449c0 con 0x7faaf8142850 2024-03-28T16:15:51.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faafebce640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:51.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faafebce640 10 monclient: _renew_subs 2024-03-28T16:15:51.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faafebce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:51.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faafebce640 1 -- 172.21.15.7:0/300693767 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf8145440 con 0x7faaf8142850 2024-03-28T16:15:51.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.940+0000 7faafebce640 1 librados: init done 2024-03-28T16:15:51.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.941+0000 7faafebce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.941+0000 7faafebce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.941+0000 7faafebce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:51.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.941+0000 7faafebce640 1 -- 172.21.15.7:0/300693767 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf8144c20 con 0x7faaf8142850 2024-03-28T16:15:51.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.948+0000 7f1696ffd640 1 -- 172.21.15.7:0/2557141569 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1690014090 con 0x7f16a00b32e0 2024-03-28T16:15:51.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.948+0000 7f1696ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-28T16:15:51.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.948+0000 7f1696ffd640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:51.949 INFO:teuthology.orchestra.run.smithi007.stdout:64424509446 2024-03-28T16:15:51.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.950+0000 7f16a84db640 10 librados: watch_flush enter 2024-03-28T16:15:51.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.950+0000 7f16a84db640 10 librados: watch_flush exit 2024-03-28T16:15:51.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.950+0000 7f16a84db640 20 client.4286.objecter shutdown clearing up homeless session... 2024-03-28T16:15:51.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.950+0000 7f16a84db640 10 client.4286.objecter successfully canceled tick 2024-03-28T16:15:51.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.951+0000 7f16a84db640 1 -- 172.21.15.7:0/2557141569 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f168406c5e0 msgr2=0x7f168406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.951+0000 7f16a84db640 1 --2- 172.21.15.7:0/2557141569 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f168406c5e0 0x7f168406eaa0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f169c0094a0 tx=0x7f169c0093f0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.951+0000 7f16a84db640 10 monclient: shutdown 2024-03-28T16:15:51.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.951+0000 7f16a84db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:51.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.951+0000 7f16a84db640 1 -- 172.21.15.7:0/2557141569 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16a00b32e0 msgr2=0x7f16a00afd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.951+0000 7f16a84db640 1 --2- 172.21.15.7:0/2557141569 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16a00b32e0 0x7f16a00afd70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f169000b4d0 tx=0x7f169000b990 comp rx=0 tx=0).stop 2024-03-28T16:15:51.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.951+0000 7f16a84db640 1 -- 172.21.15.7:0/2557141569 shutdown_connections 2024-03-28T16:15:51.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.951+0000 7f16a84db640 1 --2- 172.21.15.7:0/2557141569 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f168406c5e0 0x7f168406eaa0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.951+0000 7f16a84db640 1 --2- 172.21.15.7:0/2557141569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16a013ee50 0x7f16a00b02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.951+0000 7f16a84db640 1 --2- 172.21.15.7:0/2557141569 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16a00b32e0 0x7f16a00afd70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.951+0000 7f16a84db640 1 --2- 172.21.15.7:0/2557141569 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16a0071480 0x7f16a013fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.951+0000 7f16a84db640 1 -- 172.21.15.7:0/2557141569 >> 172.21.15.7:0/2557141569 conn(0x7f16a00b70e0 msgr2=0x7f16a0139950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:51.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.951+0000 7f16a84db640 1 -- 172.21.15.7:0/2557141569 shutdown_connections 2024-03-28T16:15:51.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.951+0000 7f16a84db640 1 -- 172.21.15.7:0/2557141569 wait complete. 2024-03-28T16:15:51.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.951+0000 7f16a84db640 1 librados: shutdown 2024-03-28T16:15:51.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.951+0000 7ffbf9046640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-28T16:15:51.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.951+0000 7ffbf9046640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-28T16:15:51.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.951+0000 7ffbf9046640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:51.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.951+0000 7ffbf9046640 1 -- 172.21.15.7:0/3396548066 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ffbf413a3c0 con 0x7ffbf40b32e0 2024-03-28T16:15:51.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.958+0000 7f1ae7fff640 1 -- 172.21.15.7:0/337397591 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1af8055750 con 0x7f1b0013ee50 2024-03-28T16:15:51.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.958+0000 7f1ae7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:51.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.958+0000 7f1ae7fff640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:51.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.962+0000 7ffbdb7fe640 1 -- 172.21.15.7:0/3396548066 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ffbe8053fd0 con 0x7ffbf40b32e0 2024-03-28T16:15:51.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.962+0000 7ffbdb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-28T16:15:51.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.962+0000 7ffbdb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:51.963 INFO:teuthology.orchestra.run.smithi007.stdout:73014444037 2024-03-28T16:15:51.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.965+0000 7ffbf9046640 10 librados: watch_flush enter 2024-03-28T16:15:51.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.965+0000 7ffbf9046640 10 librados: watch_flush exit 2024-03-28T16:15:51.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.965+0000 7ffbf9046640 20 client.4335.objecter shutdown clearing up homeless session... 2024-03-28T16:15:51.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.965+0000 7ffbf9046640 10 client.4335.objecter successfully canceled tick 2024-03-28T16:15:51.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.965+0000 7ffbf9046640 1 -- 172.21.15.7:0/3396548066 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7ffbe006c5f0 msgr2=0x7ffbe006eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.966+0000 7ffbf9046640 1 --2- 172.21.15.7:0/3396548066 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7ffbe006c5f0 0x7ffbe006eab0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ffbe40098f0 tx=0x7ffbe4008040 comp rx=0 tx=0).stop 2024-03-28T16:15:51.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.966+0000 7ffbf9046640 10 monclient: shutdown 2024-03-28T16:15:51.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.966+0000 7ffbf9046640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:51.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.966+0000 7ffbf9046640 1 -- 172.21.15.7:0/3396548066 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbf40b32e0 msgr2=0x7ffbf413c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:51.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.966+0000 7ffbf9046640 1 --2- 172.21.15.7:0/3396548066 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbf40b32e0 0x7ffbf413c3a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ffbe800c400 tx=0x7ffbe800c8c0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.966+0000 7ffbf9046640 1 -- 172.21.15.7:0/3396548066 shutdown_connections 2024-03-28T16:15:51.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.966+0000 7ffbf9046640 1 --2- 172.21.15.7:0/3396548066 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7ffbe006c5f0 0x7ffbe006eab0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.966+0000 7ffbf9046640 1 --2- 172.21.15.7:0/3396548066 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffbf40b32e0 0x7ffbf413c3a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.966+0000 7ffbf9046640 1 --2- 172.21.15.7:0/3396548066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffbf413ee50 0x7ffbf4142810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.966+0000 7ffbf9046640 1 --2- 172.21.15.7:0/3396548066 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffbf4071480 0x7ffbf4133ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:51.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.966+0000 7ffbf9046640 1 -- 172.21.15.7:0/3396548066 >> 172.21.15.7:0/3396548066 conn(0x7ffbf40b70e0 msgr2=0x7ffbf40b4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:51.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.967+0000 7ffbf9046640 1 -- 172.21.15.7:0/3396548066 shutdown_connections 2024-03-28T16:15:51.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.967+0000 7ffbf9046640 1 -- 172.21.15.7:0/3396548066 wait complete. 2024-03-28T16:15:51.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:51.967+0000 7ffbf9046640 1 librados: shutdown 2024-03-28T16:15:51.970 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.7 2024-03-28T16:15:51.984 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444038 got 73014444037 for osd.1 2024-03-28T16:15:52.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.012+0000 7f9d680af640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-28T16:15:52.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.012+0000 7f9d680af640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-28T16:15:52.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.012+0000 7f9d680af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:52.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.012+0000 7f9d680af640 1 -- 172.21.15.7:0/2093165412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f9d60067040 con 0x7f9d6013ee50 2024-03-28T16:15:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.023+0000 7fe79d020640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-28T16:15:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.023+0000 7fe79d020640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-28T16:15:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.023+0000 7fe79d020640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.023+0000 7fe79d020640 1 -- 172.21.15.7:0/3116337946 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fe790029fa0 con 0x7fe790032ec0 2024-03-28T16:15:52.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.029+0000 7fe794ff9640 1 -- 172.21.15.7:0/3116337946 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe790029fa0 con 0x7fe790032ec0 2024-03-28T16:15:52.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.029+0000 7fe794ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-28T16:15:52.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.029+0000 7fe794ff9640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:52.030 INFO:teuthology.orchestra.run.smithi007.stdout:98784247814 2024-03-28T16:15:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.031+0000 7fe79d020640 10 librados: watch_flush enter 2024-03-28T16:15:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.031+0000 7fe79d020640 10 librados: watch_flush exit 2024-03-28T16:15:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.031+0000 7fe79d020640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-28T16:15:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.031+0000 7fe79d020640 10 client.4292.objecter successfully canceled tick 2024-03-28T16:15:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.031+0000 7fe79d020640 1 -- 172.21.15.7:0/3116337946 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fe77c06c5e0 msgr2=0x7fe77c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.031+0000 7fe79d020640 1 --2- 172.21.15.7:0/3116337946 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fe77c06c5e0 0x7fe77c06eaa0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe78c009890 tx=0x7fe78c008040 comp rx=0 tx=0).stop 2024-03-28T16:15:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.031+0000 7fe79d020640 10 monclient: shutdown 2024-03-28T16:15:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.031+0000 7fe79d020640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.031+0000 7fe79d020640 1 -- 172.21.15.7:0/3116337946 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe790032ec0 msgr2=0x7fe7900b7610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.031+0000 7fe79d020640 1 --2- 172.21.15.7:0/3116337946 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe790032ec0 0x7fe7900b7610 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe7840090b0 tx=0x7fe7840046c0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.031+0000 7fe79d020640 1 -- 172.21.15.7:0/3116337946 shutdown_connections 2024-03-28T16:15:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.031+0000 7fe79d020640 1 --2- 172.21.15.7:0/3116337946 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fe77c06c5e0 0x7fe77c06eaa0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.031+0000 7fe79d020640 1 --2- 172.21.15.7:0/3116337946 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe79002ec10 0x7fe7900cb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.031+0000 7fe79d020640 1 --2- 172.21.15.7:0/3116337946 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe790032ec0 0x7fe7900b7610 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.031+0000 7fe79d020640 1 --2- 172.21.15.7:0/3116337946 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe79002e1d0 0x7fe7900caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.031+0000 7fe79d020640 1 -- 172.21.15.7:0/3116337946 >> 172.21.15.7:0/3116337946 conn(0x7fe790036610 msgr2=0x7fe7900d6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.031+0000 7fe79d020640 1 -- 172.21.15.7:0/3116337946 shutdown_connections 2024-03-28T16:15:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.031+0000 7fe79d020640 1 -- 172.21.15.7:0/3116337946 wait complete. 2024-03-28T16:15:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.031+0000 7fe79d020640 1 librados: shutdown 2024-03-28T16:15:52.043 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247815 got 98784247814 for osd.5 2024-03-28T16:15:52.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.074+0000 7f95b99df640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-28T16:15:52.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.074+0000 7f95b99df640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-28T16:15:52.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.074+0000 7f95b99df640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:52.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.074+0000 7f95b99df640 1 -- 172.21.15.7:0/3321025491 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f95ac0cbfe0 con 0x7f95ac035640 2024-03-28T16:15:52.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.109+0000 7f3fd5f37640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-28T16:15:52.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.109+0000 7f3fd5f37640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-28T16:15:52.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.109+0000 7f3fd5f37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:52.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.109+0000 7f3fd5f37640 1 -- 172.21.15.7:0/3691783127 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f3fd0139cc0 con 0x7f3fd00b32e0 2024-03-28T16:15:52.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.113+0000 7f3fb7fff640 1 -- 172.21.15.7:0/3691783127 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3fc8043090 con 0x7f3fd00b32e0 2024-03-28T16:15:52.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.113+0000 7f3fb7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-28T16:15:52.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.113+0000 7f3fb7fff640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:52.187 INFO:teuthology.orchestra.run.smithi007.stdout:60129542149 2024-03-28T16:15:52.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.115+0000 7f3fd5f37640 10 librados: watch_flush enter 2024-03-28T16:15:52.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.115+0000 7f3fd5f37640 10 librados: watch_flush exit 2024-03-28T16:15:52.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.115+0000 7f3fd5f37640 20 client.4332.objecter shutdown clearing up homeless session... 2024-03-28T16:15:52.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.115+0000 7f3fd5f37640 10 client.4332.objecter successfully canceled tick 2024-03-28T16:15:52.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.115+0000 7f3fd5f37640 1 -- 172.21.15.7:0/3691783127 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f3fbc06c610 msgr2=0x7f3fbc06ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:52.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.115+0000 7f3fd5f37640 1 --2- 172.21.15.7:0/3691783127 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f3fbc06c610 0x7f3fbc06ead0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0001560 tx=0x7f3fc0007990 comp rx=0 tx=0).stop 2024-03-28T16:15:52.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.115+0000 7f3fd5f37640 10 monclient: shutdown 2024-03-28T16:15:52.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.115+0000 7f3fd5f37640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:52.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.115+0000 7f3fd5f37640 1 -- 172.21.15.7:0/3691783127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fd00b32e0 msgr2=0x7f3fd0138380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:52.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.115+0000 7f3fd5f37640 1 --2- 172.21.15.7:0/3691783127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fd00b32e0 0x7f3fd0138380 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8036040 tx=0x7f3fc8004a40 comp rx=0 tx=0).stop 2024-03-28T16:15:52.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.115+0000 7f3fd5f37640 1 -- 172.21.15.7:0/3691783127 shutdown_connections 2024-03-28T16:15:52.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.115+0000 7f3fd5f37640 1 --2- 172.21.15.7:0/3691783127 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f3fbc06c610 0x7f3fbc06ead0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.115+0000 7f3fd5f37640 1 --2- 172.21.15.7:0/3691783127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fd00b32e0 0x7f3fd0138380 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.115+0000 7f3fd5f37640 1 --2- 172.21.15.7:0/3691783127 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3fd00a86e0 0x7f3fd01d70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.115+0000 7f3fd5f37640 1 --2- 172.21.15.7:0/3691783127 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fd0071480 0x7f3fd01d6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.115+0000 7f3fd5f37640 1 -- 172.21.15.7:0/3691783127 >> 172.21.15.7:0/3691783127 conn(0x7f3fd00b70e0 msgr2=0x7f3fd0138fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:52.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.115+0000 7f3fd5f37640 1 -- 172.21.15.7:0/3691783127 shutdown_connections 2024-03-28T16:15:52.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.115+0000 7f3fd5f37640 1 -- 172.21.15.7:0/3691783127 wait complete. 2024-03-28T16:15:52.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.115+0000 7f3fd5f37640 1 librados: shutdown 2024-03-28T16:15:52.191 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.4 2024-03-28T16:15:52.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.149+0000 7f1b05b9c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-28T16:15:52.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.149+0000 7f1b05b9c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-28T16:15:52.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.149+0000 7f1b05b9c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:52.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.149+0000 7f1b05b9c640 1 -- 172.21.15.7:0/337397591 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f1b00144750 con 0x7f1b0013ee50 2024-03-28T16:15:52.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.197+0000 7faaf57fa640 1 -- 172.21.15.7:0/300693767 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7faaf003f910 con 0x7faaf8142850 2024-03-28T16:15:52.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.198+0000 7faaf57fa640 1 --2- 172.21.15.7:0/300693767 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7faadc06c5e0 0x7faadc06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:52.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.198+0000 7faafc96a640 1 --2- 172.21.15.7:0/300693767 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7faadc06c5e0 0x7faadc06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:52.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.198+0000 7faafc96a640 10 monclient: get_auth_request con 0x7faadc06c5e0 auth_method 0 2024-03-28T16:15:52.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.199+0000 7faaf57fa640 10 client.4378.objecter ms_handle_connect 0x7faadc06c5e0 2024-03-28T16:15:52.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.199+0000 7faafc96a640 1 --2- 172.21.15.7:0/300693767 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7faadc06c5e0 0x7faadc06eaa0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7faaec001560 tx=0x7faaec007990 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:52.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.201+0000 7faaf57fa640 1 -- 172.21.15.7:0/300693767 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7faaf007b350 con 0x7faaf8142850 2024-03-28T16:15:52.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.201+0000 7faaf57fa640 10 client.4378.objecter ms_dispatch 0x7faaf8000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:52.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.201+0000 7faaf57fa640 3 client.4378.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:52.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.201+0000 7faaf57fa640 3 client.4378.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:52.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.201+0000 7faaf57fa640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-28T16:15:52.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.212+0000 7faaf57fa640 1 -- 172.21.15.7:0/300693767 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faaf0076f90 con 0x7faaf8142850 2024-03-28T16:15:52.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.212+0000 7faaf57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:52.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.212+0000 7faaf57fa640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:52.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.221+0000 7f1ae7fff640 1 -- 172.21.15.7:0/337397591 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1af8018020 con 0x7f1b0013ee50 2024-03-28T16:15:52.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.221+0000 7f1ae7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-28T16:15:52.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.221+0000 7f1ae7fff640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:52.222 INFO:teuthology.orchestra.run.smithi007.stdout:90194313226 2024-03-28T16:15:52.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.223+0000 7f1b05b9c640 10 librados: watch_flush enter 2024-03-28T16:15:52.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.223+0000 7f1b05b9c640 10 librados: watch_flush exit 2024-03-28T16:15:52.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.223+0000 7f1b05b9c640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-28T16:15:52.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.223+0000 7f1b05b9c640 10 client.4375.objecter successfully canceled tick 2024-03-28T16:15:52.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.223+0000 7f95b17fa640 1 -- 172.21.15.7:0/3321025491 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f95ac0cbfe0 con 0x7f95ac035640 2024-03-28T16:15:52.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.223+0000 7f95b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-28T16:15:52.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.223+0000 7f95b17fa640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:52.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.223+0000 7f1b05b9c640 1 -- 172.21.15.7:0/337397591 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f1aec06c590 msgr2=0x7f1aec06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:52.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.223+0000 7f1b05b9c640 1 --2- 172.21.15.7:0/337397591 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f1aec06c590 0x7f1aec06ea50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1af0035000 tx=0x7f1af0030000 comp rx=0 tx=0).stop 2024-03-28T16:15:52.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.223+0000 7f1b05b9c640 10 monclient: shutdown 2024-03-28T16:15:52.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.223+0000 7f1b05b9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:52.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.223+0000 7f1b05b9c640 1 -- 172.21.15.7:0/337397591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b0013ee50 msgr2=0x7f1b00144020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:52.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.223+0000 7f1b05b9c640 1 --2- 172.21.15.7:0/337397591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b0013ee50 0x7f1b00144020 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1af8009cb0 tx=0x7f1af800c490 comp rx=0 tx=0).stop 2024-03-28T16:15:52.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.223+0000 7f1b05b9c640 1 -- 172.21.15.7:0/337397591 shutdown_connections 2024-03-28T16:15:52.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.223+0000 7f1b05b9c640 1 --2- 172.21.15.7:0/337397591 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f1aec06c590 0x7f1aec06ea50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.223+0000 7f1b05b9c640 1 --2- 172.21.15.7:0/337397591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b000b32e0 0x7f1b00143ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.223+0000 7f1b05b9c640 1 --2- 172.21.15.7:0/337397591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b0013ee50 0x7f1b00144020 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.223+0000 7f1b05b9c640 1 --2- 172.21.15.7:0/337397591 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1b00071480 0x7f1b0013de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.223+0000 7f1b05b9c640 1 -- 172.21.15.7:0/337397591 >> 172.21.15.7:0/337397591 conn(0x7f1b000b70e0 msgr2=0x7f1b000b17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:52.227 INFO:teuthology.orchestra.run.smithi007.stdout:64424509444 2024-03-28T16:15:52.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.224+0000 7f1b05b9c640 1 -- 172.21.15.7:0/337397591 shutdown_connections 2024-03-28T16:15:52.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.224+0000 7f1b05b9c640 1 -- 172.21.15.7:0/337397591 wait complete. 2024-03-28T16:15:52.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.224+0000 7f1b05b9c640 1 librados: shutdown 2024-03-28T16:15:52.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.226+0000 7f95b99df640 10 librados: watch_flush enter 2024-03-28T16:15:52.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.226+0000 7f95b99df640 10 librados: watch_flush exit 2024-03-28T16:15:52.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.226+0000 7f95b99df640 20 client.4363.objecter shutdown clearing up homeless session... 2024-03-28T16:15:52.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.226+0000 7f95b99df640 10 client.4363.objecter successfully canceled tick 2024-03-28T16:15:52.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.226+0000 7f95b99df640 1 -- 172.21.15.7:0/3321025491 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f959806c590 msgr2=0x7f959806ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:52.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.226+0000 7f95b99df640 1 --2- 172.21.15.7:0/3321025491 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f959806c590 0x7f959806ea50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f95a8036000 tx=0x7f95a803c040 comp rx=0 tx=0).stop 2024-03-28T16:15:52.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.226+0000 7f95b99df640 10 monclient: shutdown 2024-03-28T16:15:52.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.226+0000 7f95b99df640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:52.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.226+0000 7f95b99df640 1 -- 172.21.15.7:0/3321025491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95ac035640 msgr2=0x7f95ac0c7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:52.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.226+0000 7f95b99df640 1 --2- 172.21.15.7:0/3321025491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95ac035640 0x7f95ac0c7dc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f95a000c410 tx=0x7f95a000c8d0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.226+0000 7f95b99df640 1 -- 172.21.15.7:0/3321025491 shutdown_connections 2024-03-28T16:15:52.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.226+0000 7f95b99df640 1 --2- 172.21.15.7:0/3321025491 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f959806c590 0x7f959806ea50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.226+0000 7f95b99df640 1 --2- 172.21.15.7:0/3321025491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95ac038fe0 0x7f95ac0ca070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.226+0000 7f95b99df640 1 --2- 172.21.15.7:0/3321025491 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95ac0cc600 0x7f95ac0ca5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.226+0000 7f95b99df640 1 --2- 172.21.15.7:0/3321025491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95ac035640 0x7f95ac0c7dc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.226+0000 7f95b99df640 1 -- 172.21.15.7:0/3321025491 >> 172.21.15.7:0/3321025491 conn(0x7f95ac0065d0 msgr2=0x7f95ac0c5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:52.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.226+0000 7f95b99df640 1 -- 172.21.15.7:0/3321025491 shutdown_connections 2024-03-28T16:15:52.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.226+0000 7f95b99df640 1 -- 172.21.15.7:0/3321025491 wait complete. 2024-03-28T16:15:52.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.226+0000 7f95b99df640 1 librados: shutdown 2024-03-28T16:15:52.237 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313227 got 90194313226 for osd.6 2024-03-28T16:15:52.245 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509444 for osd.0 2024-03-28T16:15:52.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.274+0000 7f9d567fc640 1 -- 172.21.15.7:0/2093165412 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9d58058650 con 0x7f9d6013ee50 2024-03-28T16:15:52.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.275+0000 7f9d567fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-28T16:15:52.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.275+0000 7f9d567fc640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:52.276 INFO:teuthology.orchestra.run.smithi007.stdout:73014444039 2024-03-28T16:15:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.276+0000 7f9d680af640 10 librados: watch_flush enter 2024-03-28T16:15:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.276+0000 7f9d680af640 10 librados: watch_flush exit 2024-03-28T16:15:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.276+0000 7f9d680af640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-28T16:15:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.276+0000 7f9d680af640 10 client.4360.objecter successfully canceled tick 2024-03-28T16:15:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.276+0000 7f9d680af640 1 -- 172.21.15.7:0/2093165412 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f9d4406c5e0 msgr2=0x7f9d4406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.276+0000 7f9d680af640 1 --2- 172.21.15.7:0/2093165412 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f9d4406c5e0 0x7f9d4406eaa0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9d50009710 tx=0x7f9d50006c90 comp rx=0 tx=0).stop 2024-03-28T16:15:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.276+0000 7f9d680af640 10 monclient: shutdown 2024-03-28T16:15:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.276+0000 7f9d680af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.276+0000 7f9d680af640 1 -- 172.21.15.7:0/2093165412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d6013ee50 msgr2=0x7f9d6013ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.276+0000 7f9d680af640 1 --2- 172.21.15.7:0/2093165412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d6013ee50 0x7f9d6013ddb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800c730 tx=0x7f9d5800cbf0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.277+0000 7f9d680af640 1 -- 172.21.15.7:0/2093165412 shutdown_connections 2024-03-28T16:15:52.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.277+0000 7f9d680af640 1 --2- 172.21.15.7:0/2093165412 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f9d4406c5e0 0x7f9d4406eaa0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.277+0000 7f9d680af640 1 --2- 172.21.15.7:0/2093165412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d6013ee50 0x7f9d6013ddb0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.277+0000 7f9d680af640 1 --2- 172.21.15.7:0/2093165412 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9d600b32e0 0x7f9d60143030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.277+0000 7f9d680af640 1 --2- 172.21.15.7:0/2093165412 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d60071480 0x7f9d601d6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.277+0000 7f9d680af640 1 -- 172.21.15.7:0/2093165412 >> 172.21.15.7:0/2093165412 conn(0x7f9d600b70e0 msgr2=0x7f9d600b4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:52.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.277+0000 7f9d680af640 1 -- 172.21.15.7:0/2093165412 shutdown_connections 2024-03-28T16:15:52.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.277+0000 7f9d680af640 1 -- 172.21.15.7:0/2093165412 wait complete. 2024-03-28T16:15:52.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.277+0000 7f9d680af640 1 librados: shutdown 2024-03-28T16:15:52.289 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444040 got 73014444039 for osd.3 2024-03-28T16:15:52.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.417+0000 7faafebce640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-28T16:15:52.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.417+0000 7faafebce640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-28T16:15:52.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.417+0000 7faafebce640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:52.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.417+0000 7faafebce640 1 -- 172.21.15.7:0/300693767 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7faaf8144c20 con 0x7faaf8142850 2024-03-28T16:15:52.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.421+0000 7faaf57fa640 1 -- 172.21.15.7:0/300693767 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faaf8144c20 con 0x7faaf8142850 2024-03-28T16:15:52.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.421+0000 7faaf57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-28T16:15:52.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.421+0000 7faaf57fa640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:52.422 INFO:teuthology.orchestra.run.smithi007.stdout:73014444040 2024-03-28T16:15:52.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.423+0000 7faafebce640 10 librados: watch_flush enter 2024-03-28T16:15:52.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.423+0000 7faafebce640 10 librados: watch_flush exit 2024-03-28T16:15:52.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.423+0000 7faafebce640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-28T16:15:52.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.423+0000 7faafebce640 10 client.4378.objecter successfully canceled tick 2024-03-28T16:15:52.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.423+0000 7faafebce640 1 -- 172.21.15.7:0/300693767 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7faadc06c5e0 msgr2=0x7faadc06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:52.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.423+0000 7faafebce640 1 --2- 172.21.15.7:0/300693767 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7faadc06c5e0 0x7faadc06eaa0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7faaec001560 tx=0x7faaec007990 comp rx=0 tx=0).stop 2024-03-28T16:15:52.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.423+0000 7faafebce640 10 monclient: shutdown 2024-03-28T16:15:52.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.423+0000 7faafebce640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:52.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.423+0000 7faafebce640 1 -- 172.21.15.7:0/300693767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faaf8142850 msgr2=0x7faaf8144f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:52.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.423+0000 7faafebce640 1 --2- 172.21.15.7:0/300693767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faaf8142850 0x7faaf8144f00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faaf0009eb0 tx=0x7faaf0004a40 comp rx=0 tx=0).stop 2024-03-28T16:15:52.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.423+0000 7faafebce640 1 -- 172.21.15.7:0/300693767 shutdown_connections 2024-03-28T16:15:52.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.423+0000 7faafebce640 1 --2- 172.21.15.7:0/300693767 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7faadc06c5e0 0x7faadc06eaa0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.423+0000 7faafebce640 1 --2- 172.21.15.7:0/300693767 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faaf813ef40 0x7faaf8141680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.423+0000 7faafebce640 1 --2- 172.21.15.7:0/300693767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faaf8142850 0x7faaf8144f00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.423+0000 7faafebce640 1 --2- 172.21.15.7:0/300693767 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faaf8071920 0x7faaf81442f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:52.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.423+0000 7faafebce640 1 -- 172.21.15.7:0/300693767 >> 172.21.15.7:0/300693767 conn(0x7faaf80b3db0 msgr2=0x7faaf8135a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:52.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.423+0000 7faafebce640 1 -- 172.21.15.7:0/300693767 shutdown_connections 2024-03-28T16:15:52.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.423+0000 7faafebce640 1 -- 172.21.15.7:0/300693767 wait complete. 2024-03-28T16:15:52.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:52.423+0000 7faafebce640 1 librados: shutdown 2024-03-28T16:15:52.436 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444040 got 73014444040 for osd.2 2024-03-28T16:15:52.436 DEBUG:teuthology.parallel:result is None 2024-03-28T16:15:52.971 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-28T16:15:52.985 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-28T16:15:53.044 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-28T16:15:53.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.093+0000 7fd3344c4640 10 monclient: get_monmap_and_config 2024-03-28T16:15:53.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.094+0000 7fd3344c4640 10 monclient: build_initial_monmap 2024-03-28T16:15:53.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.094+0000 7fd3344c4640 10 monclient: monmap: 2024-03-28T16:15:53.095 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:53.095 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.095 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:53.095630+0000 2024-03-28T16:15:53.095 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:53.095630+0000 2024-03-28T16:15:53.095 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:53.095 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.095 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:53.095 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:53.096 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:53.096 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:53.096 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:53.096 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.094+0000 7fd3344c4640 1 Processor -- start 2024-03-28T16:15:53.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.094+0000 7fd3344c4640 1 -- start start 2024-03-28T16:15:53.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.094+0000 7fd3344c4640 10 monclient: init 2024-03-28T16:15:53.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.095+0000 7fd3344c4640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:53.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.095+0000 7fd3344c4640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-03-28T16:15:53.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.095+0000 7fd3344c4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd32c071480 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:53.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.095+0000 7fd3344c4640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd32c0b32e0 0x7fd32c144a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.095+0000 7fd3344c4640 10 monclient(hunting): picked mon.noname-b con 0x7fd32c0b32e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:53.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.095+0000 7fd3344c4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd32c0a86e0 0x7fd32c144f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.095+0000 7fd3344c4640 10 monclient(hunting): picked mon.noname-a con 0x7fd32c0a86e0 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:53.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.095+0000 7fd3344c4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c0a8c90 con 0x7fd32c071480 2024-03-28T16:15:53.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.095+0000 7fd3344c4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.095+0000 7fd3344c4640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd32c0a9670 con 0x7fd32c0a86e0 2024-03-28T16:15:53.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.095+0000 7fd3344c4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.095+0000 7fd3344c4640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd32c071860 con 0x7fd32c0b32e0 2024-03-28T16:15:53.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.095+0000 7fd3344c4640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:53.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.095+0000 7fd3344c4640 10 monclient(hunting): authenticate will time out at 1420.237487s 2024-03-28T16:15:53.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.096+0000 7fd331a5f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd32c0b32e0 0x7fd32c144a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.096+0000 7fd331a5f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd32c0b32e0 0x7fd32c144a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60326/0 (socket says 172.21.15.7:60326) 2024-03-28T16:15:53.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.096+0000 7fd331a5f640 1 -- 172.21.15.7:0/2861247173 learned_addr learned my addr 172.21.15.7:0/2861247173 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:53.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.096+0000 7fd331a5f640 10 monclient(hunting): get_auth_request con 0x7fd32c0b32e0 auth_method 0 2024-03-28T16:15:53.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.096+0000 7fd331a5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:53.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.096+0000 7fd331a5f640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.096+0000 7fd331a5f640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:53.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.096+0000 7fd331a5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:53.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.096+0000 7fd331a5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:53.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.096+0000 7fd331a5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:53.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.098+0000 7fd332a61640 1 --2- 172.21.15.7:0/2861247173 >> v2:172.21.15.7:3300/0 conn(0x7fd32c0a86e0 0x7fd32c144f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.098+0000 7fd332a61640 10 monclient(hunting): get_auth_request con 0x7fd32c0a86e0 auth_method 0 2024-03-28T16:15:53.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.098+0000 7fd332a61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:53.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.098+0000 7fd332a61640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.098+0000 7fd332a61640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:53.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.098+0000 7fd331a5f640 10 monclient(hunting): handle_auth_done global_id 4298 payload 274 2024-03-28T16:15:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.098+0000 7fd331a5f640 1 -- 172.21.15.7:0/2861247173 >> v2:172.21.15.7:3300/0 conn(0x7fd32c0a86e0 msgr2=0x7fd32c144f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.098+0000 7fd331a5f640 1 --2- 172.21.15.7:0/2861247173 >> v2:172.21.15.7:3300/0 conn(0x7fd32c0a86e0 0x7fd32c144f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.098+0000 7fd331a5f640 1 -- 172.21.15.7:0/2861247173 >> v1:172.21.15.7:6789/0 conn(0x7fd32c071480 legacy=0x7fd32c13e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.098+0000 7fd331a5f640 10 monclient: _finish_hunting 0 2024-03-28T16:15:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.098+0000 7fd331a5f640 1 monclient: found mon.noname-b 2024-03-28T16:15:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.098+0000 7fd331a5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.098+0000 7fd331a5f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-03-28T16:15:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.098+0000 7fd331a5f640 1 -- 172.21.15.7:0/2861247173 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c13f020 con 0x7fd32c0b32e0 2024-03-28T16:15:53.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.101+0000 7fd331a5f640 1 --2- 172.21.15.7:0/2861247173 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd32c0b32e0 0x7fd32c144a00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd31c009870 tx=0x7fd31c02b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbe17cc07208032d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:53.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.103+0000 7fd33125e640 1 -- 172.21.15.7:0/2861247173 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31c031740 con 0x7fd32c0b32e0 2024-03-28T16:15:53.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.103+0000 7fd33125e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.103+0000 7fd33125e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:53.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.103+0000 7fd33125e640 10 monclient: dump: 2024-03-28T16:15:53.104 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.104 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.104 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.104 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.105 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.105 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.105 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.105 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.105 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.105 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.103+0000 7fd33125e640 10 monclient: _finish_auth 0 2024-03-28T16:15:53.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.103+0000 7fd33125e640 10 monclient: _check_auth_tickets 2024-03-28T16:15:53.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.103+0000 7fd33125e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:53.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.103+0000 7fd3344c4640 5 monclient: authenticate success, global_id 4298 2024-03-28T16:15:53.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.103+0000 7fd3344c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.105+0000 7fd33125e640 1 -- 172.21.15.7:0/2861247173 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd31c0318a0 con 0x7fd32c0b32e0 2024-03-28T16:15:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.105+0000 7fd33125e640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.105+0000 7fd33125e640 1 -- 172.21.15.7:0/2861247173 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31c031c30 con 0x7fd32c0b32e0 2024-03-28T16:15:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.105+0000 7fd3344c4640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.105+0000 7fd3344c4640 4 set_mon_vals no callback set 2024-03-28T16:15:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.105+0000 7fd33125e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.105+0000 7fd33125e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:15:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.105+0000 7fd33125e640 10 monclient: dump: 2024-03-28T16:15:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.106 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.107 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.107 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.107 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.107 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.107 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.107 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.107 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.106+0000 7fd3344c4640 10 monclient: shutdown 2024-03-28T16:15:53.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.106+0000 7fd3344c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.106+0000 7fd3344c4640 1 -- 172.21.15.7:0/2861247173 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd32c0b32e0 msgr2=0x7fd32c144a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.106+0000 7fd3344c4640 1 --2- 172.21.15.7:0/2861247173 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd32c0b32e0 0x7fd32c144a00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd31c009870 tx=0x7fd31c02b7e0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.106+0000 7fd3344c4640 1 -- 172.21.15.7:0/2861247173 shutdown_connections 2024-03-28T16:15:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.106+0000 7fd3344c4640 1 --2- 172.21.15.7:0/2861247173 >> v2:172.21.15.7:3300/0 conn(0x7fd32c0a86e0 0x7fd32c144f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.106+0000 7fd3344c4640 1 --2- 172.21.15.7:0/2861247173 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd32c0b32e0 0x7fd32c144a00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.106+0000 7fd3344c4640 1 -- 172.21.15.7:0/2861247173 >> 172.21.15.7:0/2861247173 conn(0x7fd32c0b70e0 msgr2=0x7fd32c0b74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.106+0000 7fd3344c4640 1 -- 172.21.15.7:0/2861247173 shutdown_connections 2024-03-28T16:15:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.106+0000 7fd3344c4640 1 -- 172.21.15.7:0/2861247173 wait complete. 2024-03-28T16:15:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.107+0000 7fd3344c4640 10 monclient: build_initial_monmap 2024-03-28T16:15:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.107+0000 7fd3344c4640 10 monclient: monmap: 2024-03-28T16:15:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:53.109 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:53.110 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.107+0000 7fd3344c4640 1 librados: starting msgr at 2024-03-28T16:15:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.107+0000 7fd3344c4640 1 librados: starting objecter 2024-03-28T16:15:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.107+0000 7fd3344c4640 1 Processor -- start 2024-03-28T16:15:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.107+0000 7fd3344c4640 1 -- start start 2024-03-28T16:15:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.107+0000 7fd3344c4640 1 librados: setting wanted keys 2024-03-28T16:15:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.107+0000 7fd3344c4640 1 librados: calling monclient init 2024-03-28T16:15:53.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.107+0000 7fd3344c4640 10 monclient: init 2024-03-28T16:15:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.108+0000 7fd3344c4640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.108+0000 7fd3344c4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-28T16:15:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.108+0000 7fd3344c4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd32c071480 0x7fd32c1381b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.108+0000 7fd3344c4640 10 monclient(hunting): picked mon.noname-b con 0x7fd32c071480 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.108+0000 7fd3344c4640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd32c0a86e0 0x7fd32c143790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.108+0000 7fd3344c4640 10 monclient(hunting): picked mon.noname-c con 0x7fd32c0a86e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.108+0000 7fd3344c4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd32c0b32e0 0x7fd32c143cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.108+0000 7fd3344c4640 10 monclient(hunting): picked mon.noname-a con 0x7fd32c0b32e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.108+0000 7fd3344c4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.108+0000 7fd3344c4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32c138ba0 con 0x7fd32c0b32e0 2024-03-28T16:15:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.108+0000 7fd3344c4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.108+0000 7fd3344c4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32c1377d0 con 0x7fd32c071480 2024-03-28T16:15:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.108+0000 7fd3344c4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.108+0000 7fd3344c4640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd32c142a30 con 0x7fd32c0a86e0 2024-03-28T16:15:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.108+0000 7fd3344c4640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.108+0000 7fd3344c4640 10 monclient(hunting): authenticate will time out at 1420.250633s 2024-03-28T16:15:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.109+0000 7fd331a5f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd32c0a86e0 0x7fd32c143790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.109+0000 7fd332260640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd32c071480 0x7fd32c1381b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.109+0000 7fd332a61640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd32c0b32e0 0x7fd32c143cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.109+0000 7fd331a5f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd32c0a86e0 0x7fd32c143790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60330/0 (socket says 172.21.15.7:60330) 2024-03-28T16:15:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.109+0000 7fd331a5f640 1 -- 172.21.15.7:0/3651179925 learned_addr learned my addr 172.21.15.7:0/3651179925 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.109+0000 7fd331a5f640 10 monclient(hunting): get_auth_request con 0x7fd32c0a86e0 auth_method 0 2024-03-28T16:15:53.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.109+0000 7fd331a5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:53.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.109+0000 7fd331a5f640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.109+0000 7fd331a5f640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:53.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.109+0000 7fd332260640 10 monclient(hunting): get_auth_request con 0x7fd32c071480 auth_method 0 2024-03-28T16:15:53.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.109+0000 7fd332260640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:53.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.109+0000 7fd332260640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.109+0000 7fd332260640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:53.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.110+0000 7fd332a61640 10 monclient(hunting): get_auth_request con 0x7fd32c0b32e0 auth_method 0 2024-03-28T16:15:53.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.110+0000 7fd332a61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:53.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.110+0000 7fd332a61640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.110+0000 7fd332a61640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:53.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.110+0000 7fd331a5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:53.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.110+0000 7fd331a5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:53.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.110+0000 7fd331a5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:53.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.110+0000 7fd332260640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:53.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.110+0000 7fd332260640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:53.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.110+0000 7fd332260640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:53.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.110+0000 7fd332a61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:53.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.110+0000 7fd332a61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:53.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.110+0000 7fd332a61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:53.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.113+0000 7fd331a5f640 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2024-03-28T16:15:53.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.113+0000 7fd331a5f640 1 -- 172.21.15.7:0/3651179925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd32c071480 msgr2=0x7fd32c1381b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.113+0000 7fd331a5f640 1 --2- 172.21.15.7:0/3651179925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd32c071480 0x7fd32c1381b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.113+0000 7fd331a5f640 1 -- 172.21.15.7:0/3651179925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd32c0b32e0 msgr2=0x7fd32c143cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.113+0000 7fd331a5f640 1 --2- 172.21.15.7:0/3651179925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd32c0b32e0 0x7fd32c143cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.113+0000 7fd331a5f640 10 monclient: _finish_hunting 0 2024-03-28T16:15:53.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.113+0000 7fd331a5f640 1 monclient: found mon.noname-c 2024-03-28T16:15:53.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.113+0000 7fd331a5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:53.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.113+0000 7fd331a5f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-03-28T16:15:53.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.113+0000 7fd331a5f640 1 -- 172.21.15.7:0/3651179925 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31c0094d0 con 0x7fd32c0a86e0 2024-03-28T16:15:53.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.113+0000 7fd332260640 1 --2- 172.21.15.7:0/3651179925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd32c071480 0x7fd32c1381b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:15:53.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.115+0000 7fd331a5f640 1 --2- 172.21.15.7:0/3651179925 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd32c0a86e0 0x7fd32c143790 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd31c000c00 tx=0x7fd31c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:53.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.115+0000 7fd322ffd640 10 client.?.objecter ms_handle_connect 0x7fd32c0a86e0 2024-03-28T16:15:53.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.115+0000 7fd322ffd640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:53.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.119+0000 7fd322ffd640 1 -- 172.21.15.7:0/3651179925 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31c033c70 con 0x7fd32c0a86e0 2024-03-28T16:15:53.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.119+0000 7fd322ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.119+0000 7fd322ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:53.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.119+0000 7fd322ffd640 10 monclient: dump: 2024-03-28T16:15:53.121 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.121 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.121 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.121 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.121 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.122 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.122 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.122 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.122 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.122 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.119+0000 7fd322ffd640 10 monclient: _finish_auth 0 2024-03-28T16:15:53.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.119+0000 7fd322ffd640 10 monclient: _check_auth_tickets 2024-03-28T16:15:53.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.119+0000 7fd322ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:53.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.119+0000 7fd322ffd640 1 -- 172.21.15.7:0/3651179925 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd31c033df0 con 0x7fd32c0a86e0 2024-03-28T16:15:53.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.119+0000 7fd3344c4640 5 monclient: authenticate success, global_id 4301 2024-03-28T16:15:53.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.119+0000 7fd3344c4640 10 monclient: _renew_subs 2024-03-28T16:15:53.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.119+0000 7fd3344c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:53.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.119+0000 7fd3344c4640 1 -- 172.21.15.7:0/3651179925 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd32c1386f0 con 0x7fd32c0a86e0 2024-03-28T16:15:53.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.119+0000 7fd322ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.119+0000 7fd330a5d640 4 set_mon_vals no callback set 2024-03-28T16:15:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.119+0000 7fd3344c4640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.120+0000 7fd3344c4640 10 monclient: _renew_subs 2024-03-28T16:15:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.120+0000 7fd322ffd640 1 -- 172.21.15.7:0/3651179925 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31c03c640 con 0x7fd32c0a86e0 2024-03-28T16:15:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.120+0000 7fd3344c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.120+0000 7fd3344c4640 1 -- 172.21.15.7:0/3651179925 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd32c1443f0 con 0x7fd32c0a86e0 2024-03-28T16:15:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.120+0000 7fd3344c4640 1 librados: init done 2024-03-28T16:15:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.120+0000 7fd322ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.120+0000 7fd322ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:15:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.120+0000 7fd322ffd640 10 monclient: dump: 2024-03-28T16:15:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.123 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.124 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.124 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.124 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.124 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.121+0000 7fd3344c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:53.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.121+0000 7fd3344c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:53.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.121+0000 7fd3344c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:53.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.121+0000 7fd3344c4640 1 -- 172.21.15.7:0/3651179925 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd32c13a1a0 con 0x7fd32c0a86e0 2024-03-28T16:15:53.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.123+0000 7f8ed788e640 10 monclient: get_monmap_and_config 2024-03-28T16:15:53.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.123+0000 7f8ed788e640 10 monclient: build_initial_monmap 2024-03-28T16:15:53.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.123+0000 7f8ed788e640 10 monclient: monmap: 2024-03-28T16:15:53.126 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:53.126 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.126 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:53.124980+0000 2024-03-28T16:15:53.126 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:53.124980+0000 2024-03-28T16:15:53.127 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:53.127 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.127 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:53.127 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:53.127 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:53.127 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:53.127 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:53.127 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.123+0000 7f8ed788e640 1 Processor -- start 2024-03-28T16:15:53.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.124+0000 7f8ed788e640 1 -- start start 2024-03-28T16:15:53.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.124+0000 7f8ed788e640 10 monclient: init 2024-03-28T16:15:53.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.124+0000 7f8ed788e640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:53.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.124+0000 7f8ed788e640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-03-28T16:15:53.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.124+0000 7f8ed788e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8ed0060970 addr v1:172.21.15.83:6789/0 2024-03-28T16:15:53.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.124+0000 7f8ed788e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ed00aee30 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.124+0000 7f8ed788e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8ed013eb00 0x7f8ed0141b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.124+0000 7f8ed788e640 10 monclient(hunting): picked mon.noname-b con 0x7f8ed013eb00 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.124+0000 7f8ed788e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed01420b0 con 0x7f8ed00aee30 2024-03-28T16:15:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.124+0000 7f8ed788e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed00b0250 con 0x7f8ed0060970 2024-03-28T16:15:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.124+0000 7f8ed788e640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.124+0000 7f8ed788e640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed0060d50 con 0x7f8ed013eb00 2024-03-28T16:15:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.124+0000 7f8ed788e640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.124+0000 7f8ed788e640 10 monclient(hunting): authenticate will time out at 1420.266829s 2024-03-28T16:15:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.125+0000 7f8ed562a640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8ed0060970 0x7f8ed013e3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.7:42432/0 (socket says 172.21.15.7:42432) 2024-03-28T16:15:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.125+0000 7f8ed4e29640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8ed00aee30 0x7f8ed0141460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54518/0 (socket says 172.21.15.7:54518) 2024-03-28T16:15:53.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.125+0000 7f8ed562a640 1 -- 172.21.15.7:0/3859525713 learned_addr learned my addr 172.21.15.7:0/3859525713 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-28T16:15:53.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.127+0000 7f8ec7fff640 1 -- 172.21.15.7:0/3859525713 <== mon.1 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275443088 0 0) 0x7f8ed00b0250 con 0x7f8ed0060970 2024-03-28T16:15:53.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.127+0000 7f8ec7fff640 10 monclient(hunting): _negotiate 2024-03-28T16:15:53.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.127+0000 7f8ec7fff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.127+0000 7f8ec7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:53.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.127+0000 7f8ec7fff640 10 monclient(hunting): my global_id is 4384 2024-03-28T16:15:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.127+0000 7f8ec7fff640 1 -- 172.21.15.7:0/3859525713 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ebc003520 con 0x7f8ed0060970 2024-03-28T16:15:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.128+0000 7f8ec7fff640 1 -- 172.21.15.7:0/3859525713 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085031327 0 0) 0x7f8ed01420b0 con 0x7f8ed00aee30 2024-03-28T16:15:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.128+0000 7f8ec7fff640 10 monclient(hunting): _negotiate 2024-03-28T16:15:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.128+0000 7f8ec7fff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.128+0000 7f8ec7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.128+0000 7f8ec7fff640 10 monclient(hunting): my global_id is 4344 2024-03-28T16:15:53.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.128+0000 7f8ec7fff640 1 -- 172.21.15.7:0/3859525713 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ebc003840 con 0x7f8ed00aee30 2024-03-28T16:15:53.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 1 -- 172.21.15.7:0/3859525713 <== mon.1 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3564087467 0 0) 0x7f8ebc003520 con 0x7f8ed0060970 2024-03-28T16:15:53.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 1 -- 172.21.15.7:0/3859525713 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8ed013eb00 msgr2=0x7f8ed0141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:53.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 1 --2- 172.21.15.7:0/3859525713 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8ed013eb00 0x7f8ed0141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 1 -- 172.21.15.7:0/3859525713 >> v1:172.21.15.7:6789/0 conn(0x7f8ed00aee30 legacy=0x7f8ed0141460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 10 monclient: _finish_hunting 0 2024-03-28T16:15:53.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 1 monclient: found mon.noname-c-legacy 2024-03-28T16:15:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.83:6789/0 2024-03-28T16:15:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 1 -- 172.21.15.7:0/3859525713 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed006e040 con 0x7f8ed0060970 2024-03-28T16:15:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 10 monclient: _finish_auth 0 2024-03-28T16:15:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 10 monclient: _check_auth_tickets 2024-03-28T16:15:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ed788e640 5 monclient: authenticate success, global_id 4384 2024-03-28T16:15:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 1 -- 172.21.15.7:0/3859525713 <== mon.1 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1105724391 0 0) 0x7f8ecc001650 con 0x7f8ed0060970 2024-03-28T16:15:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-03-28T16:15:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 10 monclient: dump: 2024-03-28T16:15:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.133 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.134 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.134 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.134 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.134 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.134 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.134 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.134 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] but i'm connected to v1:172.21.15.83:6789/0, reconnecting 2024-03-28T16:15:53.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:53.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 1 -- 172.21.15.7:0/3859525713 >> v1:172.21.15.83:6789/0 conn(0x7f8ed0060970 legacy=0x7f8ed013e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-28T16:15:53.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 1 --2- 172.21.15.7:0/3859525713 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8ed013eb00 0x7f8ebc003d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 10 monclient(hunting): picked mon.c con 0x7f8ed013eb00 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:53.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 1 --2- 172.21.15.7:0/3859525713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ebc0042a0 0x7f8ebc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 10 monclient(hunting): picked mon.b con 0x7f8ebc0042a0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:53.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 1 --2- 172.21.15.7:0/3859525713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ebc004f50 0x7f8ebc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 10 monclient(hunting): picked mon.a con 0x7f8ebc004f50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:53.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 1 -- 172.21.15.7:0/3859525713 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed0060d50 con 0x7f8ebc004f50 2024-03-28T16:15:53.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 1 -- 172.21.15.7:0/3859525713 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc00bec0 con 0x7f8ebc0042a0 2024-03-28T16:15:53.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 1 -- 172.21.15.7:0/3859525713 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc00c000 con 0x7f8ed013eb00 2024-03-28T16:15:53.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:53.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ec7fff640 10 monclient(hunting): _finish_auth 0 2024-03-28T16:15:53.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.131+0000 7f8ed788e640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:53.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.132+0000 7f8ed5e2b640 1 --2- 172.21.15.7:0/3859525713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ebc004f50 0x7f8ebc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.132+0000 7f8ed4e29640 1 --2- 172.21.15.7:0/3859525713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ebc0042a0 0x7f8ebc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.133+0000 7f8ed4e29640 10 monclient(hunting): get_auth_request con 0x7f8ebc0042a0 auth_method 0 2024-03-28T16:15:53.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.133+0000 7f8ed4e29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:53.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.133+0000 7f8ed4e29640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.133+0000 7f8ed4e29640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:15:53.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.133+0000 7f8ed4e29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:53.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.133+0000 7f8ed4e29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:53.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.133+0000 7f8ed4e29640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-28T16:15:53.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.133+0000 7f8ed5e2b640 10 monclient(hunting): get_auth_request con 0x7f8ebc004f50 auth_method 0 2024-03-28T16:15:53.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.133+0000 7f8ed5e2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:53.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.133+0000 7f8ed5e2b640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.133+0000 7f8ed5e2b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-28T16:15:53.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.134+0000 7f8ed5e2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:53.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.134+0000 7f8ed5e2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:53.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.134+0000 7f8ed5e2b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-28T16:15:53.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.135+0000 7f8ed4e29640 10 monclient(hunting): handle_auth_done global_id 4384 payload 293 2024-03-28T16:15:53.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.135+0000 7f8ed4e29640 1 -- 172.21.15.7:0/3859525713 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8ed013eb00 msgr2=0x7f8ebc003d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:53.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.135+0000 7f8ed4e29640 1 --2- 172.21.15.7:0/3859525713 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8ed013eb00 0x7f8ebc003d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.135+0000 7f8ed4e29640 1 -- 172.21.15.7:0/3859525713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ebc004f50 msgr2=0x7f8ebc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:53.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.135+0000 7f8ed4e29640 1 --2- 172.21.15.7:0/3859525713 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ebc004f50 0x7f8ebc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.135+0000 7f8ed4e29640 10 monclient: _finish_hunting 0 2024-03-28T16:15:53.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.135+0000 7f8ed4e29640 1 monclient: found mon.b 2024-03-28T16:15:53.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.135+0000 7f8ed4e29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:53.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.135+0000 7f8ed4e29640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:53.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.135+0000 7f8ed4e29640 1 -- 172.21.15.7:0/3859525713 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed006e040 con 0x7f8ebc0042a0 2024-03-28T16:15:53.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.135+0000 7f8ed4e29640 10 monclient: _finish_auth 0 2024-03-28T16:15:53.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.135+0000 7f8ed4e29640 10 monclient: _check_auth_tickets 2024-03-28T16:15:53.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.135+0000 7f8ed4e29640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:53.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.137+0000 7f8ed4e29640 1 --2- 172.21.15.7:0/3859525713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ebc0042a0 0x7f8ebc0047c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00b0100 tx=0x7f8ebc0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58983639ffc0bc3d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:53.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.141+0000 7f8ec7fff640 1 -- 172.21.15.7:0/3859525713 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0007270 con 0x7f8ebc0042a0 2024-03-28T16:15:53.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.141+0000 7f8ec7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.141+0000 7f8ec7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:53.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.141+0000 7f8ec7fff640 10 monclient: dump: 2024-03-28T16:15:53.142 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.142 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.142 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.142 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.142 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.142 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.142 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.142 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.142 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.143 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.141+0000 7f8ec7fff640 1 -- 172.21.15.7:0/3859525713 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec00073d0 con 0x7f8ebc0042a0 2024-03-28T16:15:53.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.141+0000 7f8ec7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:53.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.141+0000 7f8ec7fff640 1 -- 172.21.15.7:0/3859525713 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec00047e0 con 0x7f8ebc0042a0 2024-03-28T16:15:53.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.141+0000 7f8ed788e640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:53.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.141+0000 7f8ed788e640 4 set_mon_vals no callback set 2024-03-28T16:15:53.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.141+0000 7f8ec7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.141+0000 7f8ec7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:53.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.141+0000 7f8ec7fff640 10 monclient: dump: 2024-03-28T16:15:53.143 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.144 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.144 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.144 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.144 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.144 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.144 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.145 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.145 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.145 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.142+0000 7f8ed788e640 10 monclient: shutdown 2024-03-28T16:15:53.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.142+0000 7f8ed788e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:53.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.142+0000 7f8ed788e640 1 -- 172.21.15.7:0/3859525713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ebc0042a0 msgr2=0x7f8ebc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.142+0000 7f8ed788e640 1 --2- 172.21.15.7:0/3859525713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ebc0042a0 0x7f8ebc0047c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00b0100 tx=0x7f8ebc0057f0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.142+0000 7f8ed5e2b640 1 -- 172.21.15.7:0/3859525713 reap_dead start 2024-03-28T16:15:53.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.142+0000 7fd322ffd640 1 -- 172.21.15.7:0/3651179925 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd31c03a040 con 0x7fd32c0a86e0 2024-03-28T16:15:53.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.142+0000 7f8ed788e640 1 -- 172.21.15.7:0/3859525713 shutdown_connections 2024-03-28T16:15:53.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.142+0000 7f8ed788e640 1 -- 172.21.15.7:0/3859525713 >> 172.21.15.7:0/3859525713 conn(0x7f8ed00b3dc0 msgr2=0x7f8ed00b41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:53.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.142+0000 7fd322ffd640 1 --2- 172.21.15.7:0/3651179925 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fd31006c590 0x7fd31006ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.142+0000 7f8ed788e640 1 -- 172.21.15.7:0/3859525713 shutdown_connections 2024-03-28T16:15:53.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.142+0000 7f8ed788e640 1 -- 172.21.15.7:0/3859525713 wait complete. 2024-03-28T16:15:53.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.142+0000 7fd332260640 1 --2- 172.21.15.7:0/3651179925 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fd31006c590 0x7fd31006ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.142+0000 7fd332260640 10 monclient: get_auth_request con 0x7fd31006c590 auth_method 0 2024-03-28T16:15:53.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.143+0000 7fd332260640 1 --2- 172.21.15.7:0/3651179925 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fd31006c590 0x7fd31006ea50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd328006fd0 tx=0x7fd328009060 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:53.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.143+0000 7fd322ffd640 10 client.4301.objecter ms_handle_connect 0x7fd31006c590 2024-03-28T16:15:53.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.143+0000 7f8ed788e640 10 monclient: build_initial_monmap 2024-03-28T16:15:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.143+0000 7f8ed788e640 10 monclient: monmap: 2024-03-28T16:15:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:53.152 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.144+0000 7f8ed788e640 1 librados: starting msgr at 2024-03-28T16:15:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.144+0000 7f8ed788e640 1 librados: starting objecter 2024-03-28T16:15:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.144+0000 7f8ed788e640 1 Processor -- start 2024-03-28T16:15:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.144+0000 7f8ed788e640 1 -- start start 2024-03-28T16:15:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.144+0000 7f8ed788e640 1 librados: setting wanted keys 2024-03-28T16:15:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.144+0000 7f8ed788e640 1 librados: calling monclient init 2024-03-28T16:15:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.144+0000 7f8ed788e640 10 monclient: init 2024-03-28T16:15:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.145+0000 7f8ed788e640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.145+0000 7f8ed788e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-28T16:15:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.145+0000 7f8ed788e640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8ebc0042a0 0x7f8ed011cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.145+0000 7f8ed788e640 10 monclient(hunting): picked mon.noname-c con 0x7f8ebc0042a0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.145+0000 7f8ed788e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ed011be70 0x7f8ed0137e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.145+0000 7f8ed788e640 10 monclient(hunting): picked mon.noname-b con 0x7f8ed011be70 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.145+0000 7f8ed788e640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed0138360 0x7f8ed01d6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.145+0000 7f8ed788e640 10 monclient(hunting): picked mon.noname-a con 0x7f8ed0138360 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.145+0000 7f8ed788e640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.145+0000 7f8ed788e640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed011c440 con 0x7f8ed0138360 2024-03-28T16:15:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.145+0000 7f8ed788e640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.145+0000 7f8ed788e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed01387e0 con 0x7f8ed011be70 2024-03-28T16:15:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.145+0000 7f8ed788e640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.145+0000 7f8ed788e640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed011d100 con 0x7f8ebc0042a0 2024-03-28T16:15:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.145+0000 7f8ed788e640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.145+0000 7f8ed788e640 10 monclient(hunting): authenticate will time out at 1420.287216s 2024-03-28T16:15:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.145+0000 7f8ed4e29640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ed011be70 0x7f8ed0137e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.146+0000 7f8ed4e29640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ed011be70 0x7f8ed0137e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:58246/0 (socket says 172.21.15.7:58246) 2024-03-28T16:15:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.146+0000 7f8ed4e29640 1 -- 172.21.15.7:0/4243004799 learned_addr learned my addr 172.21.15.7:0/4243004799 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.146+0000 7f8ed4e29640 10 monclient(hunting): get_auth_request con 0x7f8ed011be70 auth_method 0 2024-03-28T16:15:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.146+0000 7f8ed4e29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.146+0000 7f8ed4e29640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.146+0000 7f8ed4e29640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:53.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.146+0000 7f8ed5e2b640 1 --2- 172.21.15.7:0/4243004799 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed0138360 0x7f8ed01d6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.146+0000 7f8ed4e29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:53.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.146+0000 7f8ed4e29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:53.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.146+0000 7f8ed4e29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:53.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.146+0000 7f8ed5e2b640 10 monclient(hunting): get_auth_request con 0x7f8ed0138360 auth_method 0 2024-03-28T16:15:53.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.146+0000 7f8ed5e2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:53.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.146+0000 7f8ed5e2b640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.146+0000 7f8ed5e2b640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:53.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.147+0000 7f8ed5e2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:53.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.147+0000 7f8ed5e2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:53.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.147+0000 7f8ed5e2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:53.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.149+0000 7f8ed4e29640 10 monclient(hunting): handle_auth_done global_id 4387 payload 824 2024-03-28T16:15:53.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.149+0000 7f8ed4e29640 1 -- 172.21.15.7:0/4243004799 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8ebc0042a0 msgr2=0x7f8ed011cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.149+0000 7f8ed4e29640 1 --2- 172.21.15.7:0/4243004799 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8ebc0042a0 0x7f8ed011cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.149+0000 7f8ed4e29640 1 -- 172.21.15.7:0/4243004799 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed0138360 msgr2=0x7f8ed01d6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.149+0000 7f8ed4e29640 1 --2- 172.21.15.7:0/4243004799 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed0138360 0x7f8ed01d6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.149+0000 7f8ed4e29640 10 monclient: _finish_hunting 0 2024-03-28T16:15:53.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.149+0000 7f8ed4e29640 1 monclient: found mon.noname-b 2024-03-28T16:15:53.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.149+0000 7f8ed4e29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:53.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.149+0000 7f8ed4e29640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:15:53.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.149+0000 7f8ed4e29640 1 -- 172.21.15.7:0/4243004799 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed00710c0 con 0x7f8ed011be70 2024-03-28T16:15:53.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.151+0000 7f8ed4e29640 1 --2- 172.21.15.7:0/4243004799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ed011be70 0x7f8ed0137e20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8ec002bd70 tx=0x7f8ec00012d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:53.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.151+0000 7f8ec5ffb640 10 client.?.objecter ms_handle_connect 0x7f8ed011be70 2024-03-28T16:15:53.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.151+0000 7f8ec5ffb640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:53.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ec5ffb640 1 -- 172.21.15.7:0/4243004799 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0007750 con 0x7f8ed011be70 2024-03-28T16:15:53.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ec5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ec5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:53.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ec5ffb640 10 monclient: dump: 2024-03-28T16:15:53.158 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.158 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.158 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.158 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.158 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.158 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.158 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.158 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.159 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.159 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ec5ffb640 10 monclient: _finish_auth 0 2024-03-28T16:15:53.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ec5ffb640 10 monclient: _check_auth_tickets 2024-03-28T16:15:53.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ec5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:53.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ed788e640 5 monclient: authenticate success, global_id 4387 2024-03-28T16:15:53.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ec5ffb640 1 -- 172.21.15.7:0/4243004799 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec003f650 con 0x7f8ed011be70 2024-03-28T16:15:53.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ec5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:53.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ec5ffb640 1 -- 172.21.15.7:0/4243004799 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec003f9c0 con 0x7f8ed011be70 2024-03-28T16:15:53.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ec5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ec5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:53.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ec5ffb640 10 monclient: dump: 2024-03-28T16:15:53.160 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.160 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.160 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.160 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.160 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.160 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.160 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.160 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.160 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.160 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ec77fe640 4 set_mon_vals no callback set 2024-03-28T16:15:53.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ed788e640 10 monclient: _renew_subs 2024-03-28T16:15:53.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ed788e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:53.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ed788e640 1 -- 172.21.15.7:0/4243004799 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed01d3fc0 con 0x7f8ed011be70 2024-03-28T16:15:53.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ed788e640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:53.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ed788e640 10 monclient: _renew_subs 2024-03-28T16:15:53.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ed788e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:53.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ed788e640 1 -- 172.21.15.7:0/4243004799 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed01d6e80 con 0x7f8ed011be70 2024-03-28T16:15:53.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.155+0000 7f8ed788e640 1 librados: init done 2024-03-28T16:15:53.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.156+0000 7f8ed788e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:53.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.156+0000 7f8ed788e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:53.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.156+0000 7f8ed788e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:53.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.156+0000 7f8ed788e640 1 -- 172.21.15.7:0/4243004799 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed0069430 con 0x7f8ed011be70 2024-03-28T16:15:53.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.171+0000 7f23b51a4640 10 monclient: get_monmap_and_config 2024-03-28T16:15:53.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.171+0000 7f23b51a4640 10 monclient: build_initial_monmap 2024-03-28T16:15:53.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.171+0000 7f23b51a4640 10 monclient: monmap: 2024-03-28T16:15:53.172 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:53.172 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.173 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:53.173181+0000 2024-03-28T16:15:53.173 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:53.173181+0000 2024-03-28T16:15:53.173 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:53.173 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.173 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:53.173 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:53.173 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:53.173 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:53.173 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:53.173 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.172+0000 7f23b51a4640 1 Processor -- start 2024-03-28T16:15:53.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.172+0000 7f23b51a4640 1 -- start start 2024-03-28T16:15:53.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.172+0000 7f23b51a4640 10 monclient: init 2024-03-28T16:15:53.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.172+0000 7f23b51a4640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:53.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.172+0000 7f23b51a4640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-03-28T16:15:53.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.172+0000 7f23b51a4640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23b0071920 0x7f23b013e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.172+0000 7f23b51a4640 10 monclient(hunting): picked mon.noname-b con 0x7f23b0071920 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:53.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.172+0000 7f23b51a4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f23b00a89a0 0x7f23b013ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.172+0000 7f23b51a4640 10 monclient(hunting): picked mon.noname-a con 0x7f23b00a89a0 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:53.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.172+0000 7f23b51a4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f23b00b1890 0x7f23b00b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.172+0000 7f23b51a4640 10 monclient(hunting): picked mon.noname-c con 0x7f23b00b1890 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:53.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.172+0000 7f23b51a4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.172+0000 7f23b51a4640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23b00aa480 con 0x7f23b00a89a0 2024-03-28T16:15:53.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.172+0000 7f23b51a4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.172+0000 7f23b51a4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23b0071d00 con 0x7f23b00b1890 2024-03-28T16:15:53.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.172+0000 7f23b51a4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.172+0000 7f23b51a4640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b00a8ec0 con 0x7f23b0071920 2024-03-28T16:15:53.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.172+0000 7f23b51a4640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:53.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.172+0000 7f23b51a4640 10 monclient(hunting): authenticate will time out at 1420.314605s 2024-03-28T16:15:53.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.173+0000 7f23ae59c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f23b00a89a0 0x7f23b013ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.173+0000 7f23ae59c640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f23b00a89a0 0x7f23b013ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38068/0 (socket says 172.21.15.7:38068) 2024-03-28T16:15:53.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.173+0000 7f23ae59c640 1 -- 172.21.15.7:0/1577935540 learned_addr learned my addr 172.21.15.7:0/1577935540 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:53.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.173+0000 7f23ae59c640 10 monclient(hunting): get_auth_request con 0x7f23b00a89a0 auth_method 0 2024-03-28T16:15:53.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.173+0000 7f23ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:53.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.173+0000 7f23ae59c640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.173+0000 7f23ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:53.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.174+0000 7f23ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:53.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.174+0000 7f23ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:53.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.174+0000 7f23ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:53.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.174+0000 7f8ec5ffb640 1 -- 172.21.15.7:0/4243004799 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8ec003fb20 con 0x7f8ed011be70 2024-03-28T16:15:53.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.175+0000 7f8ec5ffb640 1 --2- 172.21.15.7:0/4243004799 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f8eb406c590 0x7f8eb406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.175+0000 7f8ed562a640 1 --2- 172.21.15.7:0/4243004799 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f8eb406c590 0x7f8eb406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.175+0000 7f8ed562a640 10 monclient: get_auth_request con 0x7f8eb406c590 auth_method 0 2024-03-28T16:15:53.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.175+0000 7f8ec5ffb640 10 client.4387.objecter ms_handle_connect 0x7f8eb406c590 2024-03-28T16:15:53.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.175+0000 7f8ed562a640 1 --2- 172.21.15.7:0/4243004799 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f8eb406c590 0x7f8eb406ea50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc0011e0 tx=0x7f8ecc0014f0 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:53.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.176+0000 7f23ae59c640 10 monclient(hunting): handle_auth_done global_id 4350 payload 274 2024-03-28T16:15:53.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.176+0000 7f23ae59c640 1 -- 172.21.15.7:0/1577935540 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23b0071920 msgr2=0x7f23b013e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:53.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.176+0000 7f23ae59c640 1 --2- 172.21.15.7:0/1577935540 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23b0071920 0x7f23b013e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.176+0000 7f23ae59c640 1 -- 172.21.15.7:0/1577935540 >> v2:172.21.15.83:3300/0 conn(0x7f23b00b1890 msgr2=0x7f23b00b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:53.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.176+0000 7f23ae59c640 1 --2- 172.21.15.7:0/1577935540 >> v2:172.21.15.83:3300/0 conn(0x7f23b00b1890 0x7f23b00b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.176+0000 7f23ae59c640 10 monclient: _finish_hunting 0 2024-03-28T16:15:53.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.176+0000 7f23ae59c640 1 monclient: found mon.noname-a 2024-03-28T16:15:53.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.176+0000 7f23ae59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:53.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.176+0000 7f23ae59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:15:53.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.176+0000 7f23ae59c640 1 -- 172.21.15.7:0/1577935540 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b00aa660 con 0x7f23b00a89a0 2024-03-28T16:15:53.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.178+0000 7f23ae59c640 1 --2- 172.21.15.7:0/1577935540 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23b00a89a0 0x7f23b013ed70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f23b00b3490 tx=0x7f23a402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d11f98f99e73319 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:53.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.182+0000 7f23add9b640 1 -- 172.21.15.7:0/1577935540 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a4031640 con 0x7f23b00a89a0 2024-03-28T16:15:53.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.182+0000 7f23add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.182+0000 7f23add9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:15:53.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.182+0000 7f23add9b640 10 monclient: dump: 2024-03-28T16:15:53.183 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.183 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.184 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.184 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.184 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.184 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.184 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.184 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.184 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.184 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.182+0000 7f23add9b640 10 monclient: _finish_auth 0 2024-03-28T16:15:53.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.182+0000 7f23add9b640 10 monclient: _check_auth_tickets 2024-03-28T16:15:53.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.182+0000 7f23add9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:53.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.182+0000 7f23add9b640 1 -- 172.21.15.7:0/1577935540 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a40317a0 con 0x7f23b00a89a0 2024-03-28T16:15:53.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.182+0000 7f23b51a4640 5 monclient: authenticate success, global_id 4350 2024-03-28T16:15:53.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.182+0000 7f23b51a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:53.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.182+0000 7f23add9b640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.182+0000 7f23b51a4640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.182+0000 7f23add9b640 1 -- 172.21.15.7:0/1577935540 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a4031b30 con 0x7f23b00a89a0 2024-03-28T16:15:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.182+0000 7f23b51a4640 4 set_mon_vals no callback set 2024-03-28T16:15:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.182+0000 7f23add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.182+0000 7f23add9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:15:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.182+0000 7f23add9b640 10 monclient: dump: 2024-03-28T16:15:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.186 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.186 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.182+0000 7f23b51a4640 10 monclient: shutdown 2024-03-28T16:15:53.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.183+0000 7f23b51a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:53.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.183+0000 7f23b51a4640 1 -- 172.21.15.7:0/1577935540 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23b00a89a0 msgr2=0x7f23b013ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.183+0000 7f23b51a4640 1 --2- 172.21.15.7:0/1577935540 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23b00a89a0 0x7f23b013ed70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f23b00b3490 tx=0x7f23a402b8f0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.183+0000 7f23b51a4640 1 -- 172.21.15.7:0/1577935540 shutdown_connections 2024-03-28T16:15:53.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.183+0000 7f23b51a4640 1 --2- 172.21.15.7:0/1577935540 >> v2:172.21.15.83:3300/0 conn(0x7f23b00b1890 0x7f23b00b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.183+0000 7f23b51a4640 1 --2- 172.21.15.7:0/1577935540 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23b00a89a0 0x7f23b013ed70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.183+0000 7f23b51a4640 1 --2- 172.21.15.7:0/1577935540 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23b0071920 0x7f23b013e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.183+0000 7f23b51a4640 1 -- 172.21.15.7:0/1577935540 >> 172.21.15.7:0/1577935540 conn(0x7f23b00b3db0 msgr2=0x7f23b00b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:53.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.183+0000 7f23b51a4640 1 -- 172.21.15.7:0/1577935540 shutdown_connections 2024-03-28T16:15:53.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.183+0000 7f23b51a4640 1 -- 172.21.15.7:0/1577935540 wait complete. 2024-03-28T16:15:53.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.183+0000 7f23b51a4640 10 monclient: build_initial_monmap 2024-03-28T16:15:53.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.183+0000 7f23b51a4640 10 monclient: monmap: 2024-03-28T16:15:53.190 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:53.190 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:53.190 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:53.190 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:53.190 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:53.190 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.190 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:53.190 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:53.191 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:53.191 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.184+0000 7f23b51a4640 1 librados: starting msgr at 2024-03-28T16:15:53.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.184+0000 7f23b51a4640 1 librados: starting objecter 2024-03-28T16:15:53.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.184+0000 7f23b51a4640 1 Processor -- start 2024-03-28T16:15:53.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.184+0000 7f23b51a4640 1 -- start start 2024-03-28T16:15:53.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.184+0000 7f23b51a4640 1 librados: setting wanted keys 2024-03-28T16:15:53.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.184+0000 7f23b51a4640 1 librados: calling monclient init 2024-03-28T16:15:53.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.184+0000 7f23b51a4640 10 monclient: init 2024-03-28T16:15:53.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.185+0000 7f23b51a4640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:53.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.185+0000 7f23b51a4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-28T16:15:53.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.185+0000 7f23b51a4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23b0071920 0x7f23b00b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.185+0000 7f23b51a4640 10 monclient(hunting): picked mon.noname-b con 0x7f23b0071920 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:53.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.185+0000 7f23b51a4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23b00b1890 0x7f23b01443d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.185+0000 7f23b51a4640 10 monclient(hunting): picked mon.noname-a con 0x7f23b00b1890 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:53.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.185+0000 7f23b51a4640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23b0140920 0x7f23b0145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.185+0000 7f23b51a4640 10 monclient(hunting): picked mon.noname-c con 0x7f23b0140920 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:53.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.185+0000 7f23b51a4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.185+0000 7f23b51a4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b0140500 con 0x7f23b00b1890 2024-03-28T16:15:53.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.185+0000 7f23b51a4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.185+0000 7f23b51a4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b0144910 con 0x7f23b0071920 2024-03-28T16:15:53.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.185+0000 7f23b51a4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.185+0000 7f23b51a4640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b0140d90 con 0x7f23b0140920 2024-03-28T16:15:53.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.185+0000 7f23b51a4640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:53.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.185+0000 7f23b51a4640 10 monclient(hunting): authenticate will time out at 1420.326973s 2024-03-28T16:15:53.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.185+0000 7f23ae59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23b00b1890 0x7f23b01443d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.186+0000 7f23ae59c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23b00b1890 0x7f23b01443d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38084/0 (socket says 172.21.15.7:38084) 2024-03-28T16:15:53.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.186+0000 7f23ae59c640 1 -- 172.21.15.7:0/605213623 learned_addr learned my addr 172.21.15.7:0/605213623 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:53.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.186+0000 7f23ae59c640 10 monclient(hunting): get_auth_request con 0x7f23b00b1890 auth_method 0 2024-03-28T16:15:53.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.186+0000 7f23ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:53.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.186+0000 7f23ae59c640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.186+0000 7f23ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:53.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.186+0000 7f23ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:53.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.186+0000 7f23ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:53.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.186+0000 7f23ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:53.193 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-28T16:15:53.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.189+0000 7f23ae59c640 10 monclient(hunting): handle_auth_done global_id 4353 payload 824 2024-03-28T16:15:53.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.189+0000 7f23ae59c640 1 -- 172.21.15.7:0/605213623 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23b0140920 msgr2=0x7f23b0145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.189+0000 7f23ae59c640 1 --2- 172.21.15.7:0/605213623 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23b0140920 0x7f23b0145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.189+0000 7f23ae59c640 1 -- 172.21.15.7:0/605213623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23b0071920 msgr2=0x7f23b00b61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.189+0000 7f23ae59c640 1 --2- 172.21.15.7:0/605213623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23b0071920 0x7f23b00b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.189+0000 7f23ae59c640 10 monclient: _finish_hunting 0 2024-03-28T16:15:53.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.189+0000 7f23ae59c640 1 monclient: found mon.noname-a 2024-03-28T16:15:53.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.189+0000 7f23ae59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:53.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.189+0000 7f23ae59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:15:53.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.189+0000 7f23ae59c640 1 -- 172.21.15.7:0/605213623 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a40094d0 con 0x7f23b00b1890 2024-03-28T16:15:53.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.194+0000 7f23ae59c640 1 --2- 172.21.15.7:0/605213623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23b00b1890 0x7f23b01443d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f23a4009870 tx=0x7f23a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:53.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.194+0000 7f23977fe640 10 client.?.objecter ms_handle_connect 0x7f23b00b1890 2024-03-28T16:15:53.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.194+0000 7f23977fe640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:53.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23977fe640 1 -- 172.21.15.7:0/605213623 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a403ac60 con 0x7f23b00b1890 2024-03-28T16:15:53.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:15:53.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23977fe640 10 monclient: dump: 2024-03-28T16:15:53.204 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.204 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.204 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.204 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.204 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.205 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.205 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.205 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.205 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.205 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23977fe640 10 monclient: _finish_auth 0 2024-03-28T16:15:53.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23977fe640 10 monclient: _check_auth_tickets 2024-03-28T16:15:53.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:53.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23977fe640 1 -- 172.21.15.7:0/605213623 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a403adc0 con 0x7f23b00b1890 2024-03-28T16:15:53.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:53.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23977fe640 1 -- 172.21.15.7:0/605213623 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a4044630 con 0x7f23b00b1890 2024-03-28T16:15:53.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:15:53.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23977fe640 10 monclient: dump: 2024-03-28T16:15:53.206 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.206 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.206 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.206 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.206 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.206 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.206 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.206 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.206 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.206 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23b51a4640 5 monclient: authenticate success, global_id 4353 2024-03-28T16:15:53.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23b51a4640 10 monclient: _renew_subs 2024-03-28T16:15:53.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23ad59a640 4 set_mon_vals no callback set 2024-03-28T16:15:53.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23b51a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:53.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23b51a4640 1 -- 172.21.15.7:0/605213623 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b0145640 con 0x7f23b00b1890 2024-03-28T16:15:53.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23b51a4640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:53.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23b51a4640 10 monclient: _renew_subs 2024-03-28T16:15:53.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23b51a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:53.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.201+0000 7f23b51a4640 1 -- 172.21.15.7:0/605213623 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b013f530 con 0x7f23b00b1890 2024-03-28T16:15:53.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.202+0000 7f23b51a4640 1 librados: init done 2024-03-28T16:15:53.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.202+0000 7f23b51a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:53.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.203+0000 7f23b51a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:53.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.203+0000 7f23b51a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:53.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.203+0000 7f23b51a4640 1 -- 172.21.15.7:0/605213623 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b0069430 con 0x7f23b00b1890 2024-03-28T16:15:53.238 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-28T16:15:53.246 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-28T16:15:53.290 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-28T16:15:53.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.297+0000 7fb72d9ab640 10 monclient: get_monmap_and_config 2024-03-28T16:15:53.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.297+0000 7fb72d9ab640 10 monclient: build_initial_monmap 2024-03-28T16:15:53.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.297+0000 7fb72d9ab640 10 monclient: monmap: 2024-03-28T16:15:53.299 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:53.299 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.299 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:53.299260+0000 2024-03-28T16:15:53.299 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:53.299260+0000 2024-03-28T16:15:53.300 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:53.300 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.301 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:53.301 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:53.302 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:53.302 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:53.302 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:53.302 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.298+0000 7fb72d9ab640 1 Processor -- start 2024-03-28T16:15:53.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.298+0000 7fb72d9ab640 1 -- start start 2024-03-28T16:15:53.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.298+0000 7fb72d9ab640 10 monclient: init 2024-03-28T16:15:53.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.298+0000 7fb72d9ab640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:53.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.298+0000 7fb72d9ab640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-03-28T16:15:53.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.298+0000 7fb72d9ab640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb728071480 0x7fb72813e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.298+0000 7fb72d9ab640 10 monclient(hunting): picked mon.noname-b con 0x7fb728071480 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:53.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.298+0000 7fb72d9ab640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb7280b32e0 0x7fb728144a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.299+0000 7fb72d9ab640 10 monclient(hunting): picked mon.noname-c con 0x7fb7280b32e0 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:53.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.299+0000 7fb72d9ab640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb72813ee50 0x7fb728144f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.299+0000 7fb72d9ab640 10 monclient(hunting): picked mon.noname-a con 0x7fb72813ee50 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:53.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.299+0000 7fb72d9ab640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.299+0000 7fb72d9ab640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb728071860 con 0x7fb72813ee50 2024-03-28T16:15:53.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.299+0000 7fb72d9ab640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.299+0000 7fb72d9ab640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7280a8c90 con 0x7fb7280b32e0 2024-03-28T16:15:53.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.299+0000 7fb72d9ab640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.299+0000 7fb72d9ab640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb728072140 con 0x7fb728071480 2024-03-28T16:15:53.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.299+0000 7fb72d9ab640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:53.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.299+0000 7fb72d9ab640 10 monclient(hunting): authenticate will time out at 1420.440998s 2024-03-28T16:15:53.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.352+0000 7f223d7e4640 10 monclient: get_monmap_and_config 2024-03-28T16:15:53.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.352+0000 7f223d7e4640 10 monclient: build_initial_monmap 2024-03-28T16:15:53.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.352+0000 7f223d7e4640 10 monclient: monmap: 2024-03-28T16:15:53.356 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:53.356 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.356 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:53.354502+0000 2024-03-28T16:15:53.356 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:53.354502+0000 2024-03-28T16:15:53.356 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:53.356 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.356 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:53.356 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:53.356 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:53.357 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:53.357 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:53.357 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.353+0000 7f223d7e4640 1 Processor -- start 2024-03-28T16:15:53.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.353+0000 7f223d7e4640 1 -- start start 2024-03-28T16:15:53.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.353+0000 7f223d7e4640 10 monclient: init 2024-03-28T16:15:53.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.354+0000 7f223d7e4640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:53.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.354+0000 7f223d7e4640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-03-28T16:15:53.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.354+0000 7f223d7e4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2238071480 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:53.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.354+0000 7f223d7e4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f22380b32e0 addr v1:172.21.15.83:6789/0 2024-03-28T16:15:53.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.354+0000 7f223d7e4640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f22380a86e0 0x7f2238145110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.354+0000 7f223d7e4640 10 monclient(hunting): picked mon.noname-a con 0x7f22380a86e0 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:53.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.354+0000 7f223d7e4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22380a8c90 con 0x7f2238071480 2024-03-28T16:15:53.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.354+0000 7f223d7e4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2238145650 con 0x7f22380b32e0 2024-03-28T16:15:53.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.354+0000 7f223d7e4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.354+0000 7f223d7e4640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2238071860 con 0x7f22380a86e0 2024-03-28T16:15:53.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.354+0000 7f223d7e4640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:53.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.354+0000 7f223d7e4640 10 monclient(hunting): authenticate will time out at 1420.496359s 2024-03-28T16:15:53.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.369+0000 7fd322ffd640 1 -- 172.21.15.7:0/3651179925 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd31c031eb0 con 0x7fd32c0a86e0 2024-03-28T16:15:53.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.369+0000 7fd322ffd640 10 client.4301.objecter ms_dispatch 0x7fd32c000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:53.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.369+0000 7fd322ffd640 3 client.4301.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:53.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.369+0000 7fd322ffd640 3 client.4301.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:53.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.369+0000 7fd322ffd640 20 client.4301.objecter dump_active .. 0 homeless 2024-03-28T16:15:53.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.399+0000 7f8ec5ffb640 1 -- 172.21.15.7:0/4243004799 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8ec007b8b0 con 0x7f8ed011be70 2024-03-28T16:15:53.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.399+0000 7f8ec5ffb640 10 client.4387.objecter ms_dispatch 0x7f8ed0000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:53.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.399+0000 7f8ec5ffb640 3 client.4387.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:53.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.399+0000 7f8ec5ffb640 3 client.4387.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:53.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.399+0000 7f8ec5ffb640 20 client.4387.objecter dump_active .. 0 homeless 2024-03-28T16:15:53.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.400+0000 7efe144d5640 10 monclient: get_monmap_and_config 2024-03-28T16:15:53.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.400+0000 7efe144d5640 10 monclient: build_initial_monmap 2024-03-28T16:15:53.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.400+0000 7efe144d5640 10 monclient: monmap: 2024-03-28T16:15:53.402 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:53.402 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.402 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:53.401857+0000 2024-03-28T16:15:53.402 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:53.401857+0000 2024-03-28T16:15:53.402 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:53.402 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.402 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:53.402 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:53.402 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:53.402 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:53.402 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:53.402 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.400+0000 7efe144d5640 1 Processor -- start 2024-03-28T16:15:53.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.400+0000 7efe144d5640 1 -- start start 2024-03-28T16:15:53.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.400+0000 7efe144d5640 10 monclient: init 2024-03-28T16:15:53.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.401+0000 7efe144d5640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:53.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.401+0000 7efe144d5640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-03-28T16:15:53.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.401+0000 7efe144d5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efe0c071920 0x7efe0c13e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.401+0000 7efe144d5640 10 monclient(hunting): picked mon.noname-c con 0x7efe0c071920 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:53.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.401+0000 7efe144d5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efe0c0a89a0 addr v1:172.21.15.83:6789/0 2024-03-28T16:15:53.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.401+0000 7efe144d5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efe0c13ed70 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:53.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.401+0000 7efe144d5640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe0c0a8ec0 con 0x7efe0c13ed70 2024-03-28T16:15:53.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.401+0000 7efe144d5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe0c13f1a0 con 0x7efe0c0a89a0 2024-03-28T16:15:53.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.401+0000 7efe144d5640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.401+0000 7efe144d5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe0c0aa480 con 0x7efe0c071920 2024-03-28T16:15:53.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.401+0000 7efe144d5640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:53.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.401+0000 7efe144d5640 10 monclient(hunting): authenticate will time out at 1420.543568s 2024-03-28T16:15:53.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.411+0000 7f8ec5ffb640 1 -- 172.21.15.7:0/4243004799 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8ec00774f0 con 0x7f8ed011be70 2024-03-28T16:15:53.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.411+0000 7f8ec5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:53.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.411+0000 7f8ec5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:53.414 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.413+0000 7fb7267fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb7280b32e0 0x7fb728144a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.414+0000 7fb7267fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fb7280b32e0 0x7fb728144a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:58268/0 (socket says 172.21.15.7:58268) 2024-03-28T16:15:53.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.414+0000 7fb7267fc640 1 -- 172.21.15.7:0/3912368754 learned_addr learned my addr 172.21.15.7:0/3912368754 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:53.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.414+0000 7fb7267fc640 10 monclient(hunting): get_auth_request con 0x7fb7280b32e0 auth_method 0 2024-03-28T16:15:53.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.414+0000 7fb7267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.414+0000 7fb7267fc640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.414+0000 7fb7267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:53.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.417+0000 7fb7267fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:53.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.417+0000 7fb7267fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:53.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.417+0000 7fb7267fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:53.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.417+0000 7f22367fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f22380b32e0 0x7f2238144a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.7:42438/0 (socket says 172.21.15.7:42438) 2024-03-28T16:15:53.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.417+0000 7f22367fc640 1 -- 172.21.15.7:0/1417382972 learned_addr learned my addr 172.21.15.7:0/1417382972 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-28T16:15:53.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.419+0000 7fb7267fc640 10 monclient(hunting): handle_auth_done global_id 4390 payload 274 2024-03-28T16:15:53.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.419+0000 7fb7267fc640 1 -- 172.21.15.7:0/3912368754 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb728071480 msgr2=0x7fb72813e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:53.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.419+0000 7fb7267fc640 1 --2- 172.21.15.7:0/3912368754 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb728071480 0x7fb72813e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.419+0000 7fb7267fc640 1 -- 172.21.15.7:0/3912368754 >> v2:172.21.15.7:3300/0 conn(0x7fb72813ee50 msgr2=0x7fb728144f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:53.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.419+0000 7fb7267fc640 1 --2- 172.21.15.7:0/3912368754 >> v2:172.21.15.7:3300/0 conn(0x7fb72813ee50 0x7fb728144f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.419+0000 7fb7267fc640 10 monclient: _finish_hunting 0 2024-03-28T16:15:53.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.419+0000 7fb7267fc640 1 monclient: found mon.noname-c 2024-03-28T16:15:53.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.419+0000 7fb7267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:53.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.419+0000 7fb7267fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3300/0 2024-03-28T16:15:53.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.419+0000 7fb7267fc640 1 -- 172.21.15.7:0/3912368754 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7280722b0 con 0x7fb7280b32e0 2024-03-28T16:15:53.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.420+0000 7efe12271640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efe0c071920 0x7efe0c13e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.423+0000 7fb7267fc640 1 --2- 172.21.15.7:0/3912368754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7280b32e0 0x7fb728144a00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb728071af0 tx=0x7fb71c02b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0f8710d29ad0710 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:53.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.424+0000 7fd322ffd640 1 -- 172.21.15.7:0/3651179925 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd31c076560 con 0x7fd32c0a86e0 2024-03-28T16:15:53.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.424+0000 7fd322ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:53.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.424+0000 7fd322ffd640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:53.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.425+0000 7efe12271640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7efe0c071920 0x7efe0c13e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:58274/0 (socket says 172.21.15.7:58274) 2024-03-28T16:15:53.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.425+0000 7efe12271640 1 -- 172.21.15.7:0/946982322 learned_addr learned my addr 172.21.15.7:0/946982322 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:53.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.425+0000 7efe12271640 10 monclient(hunting): get_auth_request con 0x7efe0c071920 auth_method 0 2024-03-28T16:15:53.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.425+0000 7efe12271640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:53.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.425+0000 7efe12271640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.425+0000 7efe12271640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:53.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.426+0000 7efe12271640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:53.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.426+0000 7efe12271640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:53.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.426+0000 7efe12271640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:53.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.428+0000 7efe12271640 10 monclient(hunting): handle_auth_done global_id 4393 payload 274 2024-03-28T16:15:53.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.428+0000 7efe12271640 1 -- 172.21.15.7:0/946982322 >> v1:172.21.15.83:6789/0 conn(0x7efe0c0a89a0 legacy=0x7efe0c0b1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:53.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.428+0000 7efe12271640 1 -- 172.21.15.7:0/946982322 >> v1:172.21.15.7:6789/0 conn(0x7efe0c13ed70 legacy=0x7efe0c14bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:53.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.428+0000 7efe12271640 10 monclient: _finish_hunting 0 2024-03-28T16:15:53.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.428+0000 7efe12271640 1 monclient: found mon.noname-c 2024-03-28T16:15:53.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.428+0000 7efe12271640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:53.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.428+0000 7efe12271640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3300/0 2024-03-28T16:15:53.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.428+0000 7efe12271640 1 -- 172.21.15.7:0/946982322 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0c0aa660 con 0x7efe0c071920 2024-03-28T16:15:53.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.431+0000 7fb725ffb640 1 -- 172.21.15.7:0/3912368754 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71c031750 con 0x7fb7280b32e0 2024-03-28T16:15:53.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.431+0000 7fb725ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.431+0000 7fb725ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:53.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.431+0000 7fb725ffb640 10 monclient: dump: 2024-03-28T16:15:53.432 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.432 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.432 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.432 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.432 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.432 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.432 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.433 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.433 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.433 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.431+0000 7fb725ffb640 10 monclient: _finish_auth 0 2024-03-28T16:15:53.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.431+0000 7fb725ffb640 10 monclient: _check_auth_tickets 2024-03-28T16:15:53.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.431+0000 7fb725ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:53.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.431+0000 7fb72d9ab640 5 monclient: authenticate success, global_id 4390 2024-03-28T16:15:53.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.431+0000 7fb72d9ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:53.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.433+0000 7efe12271640 1 --2- 172.21.15.7:0/946982322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe0c071920 0x7efe0c13e830 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efe08009870 tx=0x7efe0802b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a00cf5d9e5900ebb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:53.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.434+0000 7fb725ffb640 1 -- 172.21.15.7:0/3912368754 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb71c0318b0 con 0x7fb7280b32e0 2024-03-28T16:15:53.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.434+0000 7fb725ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:53.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.434+0000 7fb725ffb640 1 -- 172.21.15.7:0/3912368754 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71c031c40 con 0x7fb7280b32e0 2024-03-28T16:15:53.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.434+0000 7fb72d9ab640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:53.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.434+0000 7fb72d9ab640 4 set_mon_vals no callback set 2024-03-28T16:15:53.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.434+0000 7fb725ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.434+0000 7fb725ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:53.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.434+0000 7fb725ffb640 10 monclient: dump: 2024-03-28T16:15:53.436 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.436 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.436 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.436 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.436 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.436 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.436 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.436 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.436 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.436 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.435+0000 7fb72d9ab640 10 monclient: shutdown 2024-03-28T16:15:53.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.435+0000 7fb72d9ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:53.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.435+0000 7fb72d9ab640 1 -- 172.21.15.7:0/3912368754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7280b32e0 msgr2=0x7fb728144a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.435+0000 7fb72d9ab640 1 --2- 172.21.15.7:0/3912368754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7280b32e0 0x7fb728144a00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb728071af0 tx=0x7fb71c02b7f0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.435+0000 7fb72d9ab640 1 -- 172.21.15.7:0/3912368754 shutdown_connections 2024-03-28T16:15:53.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.435+0000 7fb72d9ab640 1 --2- 172.21.15.7:0/3912368754 >> v2:172.21.15.7:3300/0 conn(0x7fb72813ee50 0x7fb728144f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.435+0000 7fb72d9ab640 1 --2- 172.21.15.7:0/3912368754 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7280b32e0 0x7fb728144a00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.435+0000 7fb72d9ab640 1 --2- 172.21.15.7:0/3912368754 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb728071480 0x7fb72813e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.435+0000 7fb72d9ab640 1 -- 172.21.15.7:0/3912368754 >> 172.21.15.7:0/3912368754 conn(0x7fb7280b70e0 msgr2=0x7fb7280b74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:53.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.435+0000 7fb72d9ab640 1 -- 172.21.15.7:0/3912368754 shutdown_connections 2024-03-28T16:15:53.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.435+0000 7fb72d9ab640 1 -- 172.21.15.7:0/3912368754 wait complete. 2024-03-28T16:15:53.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.436+0000 7fb72d9ab640 10 monclient: build_initial_monmap 2024-03-28T16:15:53.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.436+0000 7fb72d9ab640 10 monclient: monmap: 2024-03-28T16:15:53.441 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:53.441 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:53.441 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:53.441 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:53.441 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:53.441 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.441 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:53.441 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:53.441 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:53.442 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.436+0000 7fb72d9ab640 1 librados: starting msgr at 2024-03-28T16:15:53.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.436+0000 7fb72d9ab640 1 librados: starting objecter 2024-03-28T16:15:53.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.436+0000 7fb72d9ab640 1 Processor -- start 2024-03-28T16:15:53.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.436+0000 7fb72d9ab640 1 -- start start 2024-03-28T16:15:53.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.436+0000 7fb72d9ab640 1 librados: setting wanted keys 2024-03-28T16:15:53.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.436+0000 7fb72d9ab640 1 librados: calling monclient init 2024-03-28T16:15:53.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.436+0000 7fb72d9ab640 10 monclient: init 2024-03-28T16:15:53.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.437+0000 7fb72d9ab640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:53.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.437+0000 7fb72d9ab640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-28T16:15:53.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.437+0000 7fb72d9ab640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb728071480 0x7fb72813faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.437+0000 7fb72d9ab640 10 monclient(hunting): picked mon.noname-c con 0x7fb728071480 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:53.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.437+0000 7fb72d9ab640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7280b32e0 0x7fb728140710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.437+0000 7fb72d9ab640 10 monclient(hunting): picked mon.noname-a con 0x7fb7280b32e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:53.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.437+0000 7fb72d9ab640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb72813ee50 0x7fb7281d68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.437+0000 7fb72d9ab640 10 monclient(hunting): picked mon.noname-b con 0x7fb72813ee50 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:53.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.437+0000 7fb72d9ab640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.437+0000 7fb72d9ab640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb728140ec0 con 0x7fb7280b32e0 2024-03-28T16:15:53.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.437+0000 7fb72d9ab640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.437+0000 7fb72d9ab640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb728140c50 con 0x7fb72813ee50 2024-03-28T16:15:53.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.437+0000 7fb72d9ab640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.437+0000 7fb72d9ab640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72813c3a0 con 0x7fb728071480 2024-03-28T16:15:53.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.437+0000 7fb72d9ab640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:53.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.437+0000 7fb72d9ab640 10 monclient(hunting): authenticate will time out at 1420.579513s 2024-03-28T16:15:53.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.438+0000 7fb726ffd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb728071480 0x7fb72813faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.438+0000 7fb726ffd640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb728071480 0x7fb72813faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60380/0 (socket says 172.21.15.7:60380) 2024-03-28T16:15:53.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.438+0000 7fb726ffd640 1 -- 172.21.15.7:0/446609966 learned_addr learned my addr 172.21.15.7:0/446609966 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:53.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.438+0000 7fb726ffd640 10 monclient(hunting): get_auth_request con 0x7fb728071480 auth_method 0 2024-03-28T16:15:53.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.438+0000 7fb726ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:53.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.438+0000 7fb726ffd640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.438+0000 7fb726ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:53.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.439+0000 7efe1126f640 1 -- 172.21.15.7:0/946982322 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe080317e0 con 0x7efe0c071920 2024-03-28T16:15:53.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.439+0000 7efe1126f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.439+0000 7efe1126f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:53.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.439+0000 7efe1126f640 10 monclient: dump: 2024-03-28T16:15:53.446 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.446 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.446 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.446 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.446 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.446 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.446 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.446 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.446 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.447 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.439+0000 7efe1126f640 10 monclient: _finish_auth 0 2024-03-28T16:15:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.439+0000 7efe1126f640 10 monclient: _check_auth_tickets 2024-03-28T16:15:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.439+0000 7efe1126f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.439+0000 7efe1126f640 1 -- 172.21.15.7:0/946982322 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe08031940 con 0x7efe0c071920 2024-03-28T16:15:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.439+0000 7efe1126f640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.439+0000 7efe1126f640 1 -- 172.21.15.7:0/946982322 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe08031cd0 con 0x7efe0c071920 2024-03-28T16:15:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.439+0000 7efe1126f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.439+0000 7efe1126f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.439+0000 7efe1126f640 10 monclient: dump: 2024-03-28T16:15:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.447 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.448 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.448 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.448 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.448 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.448 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.439+0000 7fb726ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:53.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.439+0000 7fb726ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:53.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.439+0000 7fb726ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:53.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.439+0000 7fb7277fe640 1 --2- 172.21.15.7:0/446609966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb72813ee50 0x7fb7281d68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.439+0000 7fb7277fe640 10 monclient(hunting): get_auth_request con 0x7fb72813ee50 auth_method 0 2024-03-28T16:15:53.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.439+0000 7fb7277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:53.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.439+0000 7fb7277fe640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.439+0000 7fb7277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:53.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.440+0000 7efe144d5640 5 monclient: authenticate success, global_id 4393 2024-03-28T16:15:53.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.440+0000 7efe144d5640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:53.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.440+0000 7efe144d5640 4 set_mon_vals no callback set 2024-03-28T16:15:53.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.441+0000 7efe144d5640 10 monclient: shutdown 2024-03-28T16:15:53.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.441+0000 7efe144d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:53.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.441+0000 7efe144d5640 1 -- 172.21.15.7:0/946982322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe0c071920 msgr2=0x7efe0c13e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.441+0000 7efe144d5640 1 --2- 172.21.15.7:0/946982322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe0c071920 0x7efe0c13e830 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efe08009870 tx=0x7efe0802b880 comp rx=0 tx=0).stop 2024-03-28T16:15:53.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.441+0000 7fb7277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:53.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.441+0000 7fb7277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:53.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.441+0000 7fb7277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:53.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.442+0000 7efe144d5640 1 -- 172.21.15.7:0/946982322 shutdown_connections 2024-03-28T16:15:53.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.442+0000 7efe144d5640 1 --2- 172.21.15.7:0/946982322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe0c071920 0x7efe0c13e830 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.442+0000 7efe144d5640 1 -- 172.21.15.7:0/946982322 >> 172.21.15.7:0/946982322 conn(0x7efe0c0b3db0 msgr2=0x7efe0c0b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:53.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.442+0000 7efe144d5640 1 -- 172.21.15.7:0/946982322 shutdown_connections 2024-03-28T16:15:53.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.442+0000 7f0627f20640 10 monclient: get_monmap_and_config 2024-03-28T16:15:53.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.442+0000 7f0627f20640 10 monclient: build_initial_monmap 2024-03-28T16:15:53.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.442+0000 7f0627f20640 10 monclient: monmap: 2024-03-28T16:15:53.453 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:53.453 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.453 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:53.443968+0000 2024-03-28T16:15:53.453 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:53.443968+0000 2024-03-28T16:15:53.453 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:53.453 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.453 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:53.453 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:53.453 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:53.453 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:53.453 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:53.453 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.442+0000 7efe144d5640 1 -- 172.21.15.7:0/946982322 wait complete. 2024-03-28T16:15:53.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.442+0000 7f0627f20640 1 Processor -- start 2024-03-28T16:15:53.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.443+0000 7f0627f20640 1 -- start start 2024-03-28T16:15:53.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.443+0000 7f0627f20640 10 monclient: init 2024-03-28T16:15:53.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.443+0000 7efe144d5640 10 monclient: build_initial_monmap 2024-03-28T16:15:53.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.443+0000 7efe144d5640 10 monclient: monmap: 2024-03-28T16:15:53.455 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:53.455 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:53.455 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:53.455 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:53.455 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:53.455 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.455 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:53.455 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:53.456 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:53.456 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.443+0000 7fb726ffd640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-03-28T16:15:53.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.443+0000 7fb726ffd640 1 -- 172.21.15.7:0/446609966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb72813ee50 msgr2=0x7fb7281d68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.443+0000 7fb726ffd640 1 --2- 172.21.15.7:0/446609966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb72813ee50 0x7fb7281d68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.443+0000 7fb726ffd640 1 -- 172.21.15.7:0/446609966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7280b32e0 msgr2=0x7fb728140710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.443+0000 7fb726ffd640 1 --2- 172.21.15.7:0/446609966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7280b32e0 0x7fb728140710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.443+0000 7fb726ffd640 10 monclient: _finish_hunting 0 2024-03-28T16:15:53.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.443+0000 7fb726ffd640 1 monclient: found mon.noname-c 2024-03-28T16:15:53.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.443+0000 7fb726ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:53.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.443+0000 7fb726ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-03-28T16:15:53.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.443+0000 7fb726ffd640 1 -- 172.21.15.7:0/446609966 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71c0094d0 con 0x7fb728071480 2024-03-28T16:15:53.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7efe144d5640 1 librados: starting msgr at 2024-03-28T16:15:53.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7efe144d5640 1 librados: starting objecter 2024-03-28T16:15:53.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7efe144d5640 1 Processor -- start 2024-03-28T16:15:53.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7f0627f20640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:53.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7f0627f20640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-03-28T16:15:53.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7f0627f20640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0620071920 0x7f062013e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7efe144d5640 1 -- start start 2024-03-28T16:15:53.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7f0627f20640 10 monclient(hunting): picked mon.noname-a con 0x7f0620071920 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:53.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7efe144d5640 1 librados: setting wanted keys 2024-03-28T16:15:53.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7f0627f20640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f06200a89a0 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:53.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7efe144d5640 1 librados: calling monclient init 2024-03-28T16:15:53.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7efe144d5640 10 monclient: init 2024-03-28T16:15:53.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7f0627f20640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f062013ed70 0x7f06200b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7f0627f20640 10 monclient(hunting): picked mon.noname-c con 0x7f062013ed70 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:53.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7f0627f20640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06200a8ec0 con 0x7f06200a89a0 2024-03-28T16:15:53.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7f0627f20640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7f0627f20640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06200aa480 con 0x7f0620071920 2024-03-28T16:15:53.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7f0627f20640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7f0627f20640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0620071d00 con 0x7f062013ed70 2024-03-28T16:15:53.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7f0627f20640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:53.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7f0627f20640 10 monclient(hunting): authenticate will time out at 1420.586471s 2024-03-28T16:15:53.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.444+0000 7efe144d5640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:53.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.445+0000 7efe144d5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-28T16:15:53.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.445+0000 7efe144d5640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7efe0c071920 0x7efe0c141fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.445+0000 7efe144d5640 10 monclient(hunting): picked mon.noname-c con 0x7efe0c071920 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:53.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.445+0000 7efe144d5640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe0c13ed70 0x7efe0c13f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.445+0000 7efe144d5640 10 monclient(hunting): picked mon.noname-a con 0x7efe0c13ed70 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:53.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.445+0000 7efe144d5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe0c142850 0x7efe0c13f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.445+0000 7efe144d5640 10 monclient(hunting): picked mon.noname-b con 0x7efe0c142850 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:53.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.445+0000 7efe144d5640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.445+0000 7efe144d5640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c1439f0 con 0x7efe0c13ed70 2024-03-28T16:15:53.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.445+0000 7efe144d5640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.445+0000 7efe144d5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c1406c0 con 0x7efe0c142850 2024-03-28T16:15:53.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.445+0000 7efe144d5640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.445+0000 7efe144d5640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c0b6380 con 0x7efe0c071920 2024-03-28T16:15:53.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.445+0000 7efe144d5640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:53.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.445+0000 7efe144d5640 10 monclient(hunting): authenticate will time out at 1420.587104s 2024-03-28T16:15:53.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.446+0000 7fb726ffd640 1 --2- 172.21.15.7:0/446609966 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb728071480 0x7fb72813faf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb71800b470 tx=0x7fb71800b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:53.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.446+0000 7fb70ffff640 10 client.?.objecter ms_handle_connect 0x7fb728071480 2024-03-28T16:15:53.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.446+0000 7fb70ffff640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:53.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.447+0000 7f06264bd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f062013ed70 0x7f06200b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.448+0000 7f06264bd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f062013ed70 0x7f06200b1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:58294/0 (socket says 172.21.15.7:58294) 2024-03-28T16:15:53.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.448+0000 7f06264bd640 1 -- 172.21.15.7:0/3196326491 learned_addr learned my addr 172.21.15.7:0/3196326491 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:53.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.448+0000 7f06264bd640 10 monclient(hunting): get_auth_request con 0x7f062013ed70 auth_method 0 2024-03-28T16:15:53.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.448+0000 7f06264bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:53.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.448+0000 7f06264bd640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.448+0000 7f06264bd640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:53.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.448+0000 7efe12271640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7efe0c071920 0x7efe0c141fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.449+0000 7efe12a72640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe0c142850 0x7efe0c13f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.449+0000 7efe12a72640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe0c142850 0x7efe0c13f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:58302/0 (socket says 172.21.15.7:58302) 2024-03-28T16:15:53.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.449+0000 7efe12a72640 1 -- 172.21.15.7:0/3709483386 learned_addr learned my addr 172.21.15.7:0/3709483386 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:53.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.449+0000 7efe12a72640 10 monclient(hunting): get_auth_request con 0x7efe0c142850 auth_method 0 2024-03-28T16:15:53.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.449+0000 7efe12a72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:53.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.449+0000 7efe12a72640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.449+0000 7efe12a72640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:53.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.449+0000 7f06264bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:53.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.449+0000 7f06264bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:53.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.449+0000 7f06264bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:53.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.449+0000 7efe12271640 10 monclient(hunting): get_auth_request con 0x7efe0c071920 auth_method 0 2024-03-28T16:15:53.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.449+0000 7efe12271640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:53.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.449+0000 7efe12271640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.449+0000 7efe12271640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:53.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.450+0000 7efe12271640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:53.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.450+0000 7efe12271640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:53.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.450+0000 7efe12271640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:53.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.450+0000 7fb70ffff640 1 -- 172.21.15.7:0/446609966 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb718011890 con 0x7fb728071480 2024-03-28T16:15:53.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.450+0000 7fb70ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.450+0000 7fb70ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:53.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.450+0000 7fb70ffff640 10 monclient: dump: 2024-03-28T16:15:53.468 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.468 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.468 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.468 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.468 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.468 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.468 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.469 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.450+0000 7fb70ffff640 10 monclient: _finish_auth 0 2024-03-28T16:15:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.450+0000 7fb70ffff640 10 monclient: _check_auth_tickets 2024-03-28T16:15:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.450+0000 7fb70ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.450+0000 7fb70ffff640 1 -- 172.21.15.7:0/446609966 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb718011a10 con 0x7fb728071480 2024-03-28T16:15:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.450+0000 7fb70ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7fb70ffff640 1 -- 172.21.15.7:0/446609966 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb718011dc0 con 0x7fb728071480 2024-03-28T16:15:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7fb70ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7fb70ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:15:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7fb70ffff640 10 monclient: dump: 2024-03-28T16:15:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.469 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.470 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.470 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.470 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.470 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.470 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.470 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.470 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7fb72d9ab640 5 monclient: authenticate success, global_id 4304 2024-03-28T16:15:53.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7fb72d9ab640 10 monclient: _renew_subs 2024-03-28T16:15:53.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7fb72d9ab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:53.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7fb72d9ab640 1 -- 172.21.15.7:0/446609966 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb72813deb0 con 0x7fb728071480 2024-03-28T16:15:53.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7fb72d9ab640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:53.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7fb72d9ab640 10 monclient: _renew_subs 2024-03-28T16:15:53.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7fb72d9ab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:53.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7fb7257fa640 4 set_mon_vals no callback set 2024-03-28T16:15:53.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7fb72d9ab640 1 -- 172.21.15.7:0/446609966 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7281d6e00 con 0x7fb728071480 2024-03-28T16:15:53.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7fb72d9ab640 1 librados: init done 2024-03-28T16:15:53.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7f06264bd640 10 monclient(hunting): handle_auth_done global_id 4399 payload 274 2024-03-28T16:15:53.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7f06264bd640 1 -- 172.21.15.7:0/3196326491 >> v2:172.21.15.7:3300/0 conn(0x7f0620071920 msgr2=0x7f062013e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:53.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7f06264bd640 1 --2- 172.21.15.7:0/3196326491 >> v2:172.21.15.7:3300/0 conn(0x7f0620071920 0x7f062013e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7f06264bd640 1 -- 172.21.15.7:0/3196326491 >> v1:172.21.15.7:6789/0 conn(0x7f06200a89a0 legacy=0x7f06200b1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:53.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7f06264bd640 10 monclient: _finish_hunting 0 2024-03-28T16:15:53.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7f06264bd640 1 monclient: found mon.noname-c 2024-03-28T16:15:53.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7f06264bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:53.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7f06264bd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3300/0 2024-03-28T16:15:53.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7f06264bd640 1 -- 172.21.15.7:0/3196326491 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f062013f1a0 con 0x7f062013ed70 2024-03-28T16:15:53.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7efe12a72640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:53.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7efe12a72640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:53.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.451+0000 7efe12a72640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:53.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.452+0000 7fb72d9ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:53.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.452+0000 7fb72d9ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:53.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.452+0000 7fb72d9ab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:53.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.452+0000 7fb72d9ab640 1 -- 172.21.15.7:0/446609966 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7280b30c0 con 0x7fb728071480 2024-03-28T16:15:53.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.453+0000 7efe12271640 10 monclient(hunting): handle_auth_done global_id 4307 payload 824 2024-03-28T16:15:53.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.453+0000 7efe12271640 1 -- 172.21.15.7:0/3709483386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe0c142850 msgr2=0x7efe0c13f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.453+0000 7efe12271640 1 --2- 172.21.15.7:0/3709483386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe0c142850 0x7efe0c13f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.454+0000 7efe12271640 1 -- 172.21.15.7:0/3709483386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe0c13ed70 msgr2=0x7efe0c13f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.454+0000 7efe12271640 1 --2- 172.21.15.7:0/3709483386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe0c13ed70 0x7efe0c13f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.454+0000 7efe12271640 10 monclient: _finish_hunting 0 2024-03-28T16:15:53.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.454+0000 7efe12271640 1 monclient: found mon.noname-c 2024-03-28T16:15:53.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.454+0000 7efe12271640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:53.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.454+0000 7efe12271640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-03-28T16:15:53.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.454+0000 7efe12271640 1 -- 172.21.15.7:0/3709483386 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe080094d0 con 0x7efe0c071920 2024-03-28T16:15:53.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.457+0000 7f06264bd640 1 --2- 172.21.15.7:0/3196326491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f062013ed70 0x7f06200b1fa0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0620072060 tx=0x7f061802be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b525b113a684f123 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:53.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.460+0000 7f0624cba640 1 -- 172.21.15.7:0/3196326491 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061802c9f0 con 0x7f062013ed70 2024-03-28T16:15:53.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.460+0000 7f0624cba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.461+0000 7f0624cba640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:53.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.461+0000 7f0624cba640 10 monclient: dump: 2024-03-28T16:15:53.475 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.476 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.476 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.476 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.476 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.476 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.476 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.476 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.476 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.476 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.461+0000 7f0624cba640 10 monclient: _finish_auth 0 2024-03-28T16:15:53.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.461+0000 7f0624cba640 10 monclient: _check_auth_tickets 2024-03-28T16:15:53.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.461+0000 7f0624cba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:53.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.461+0000 7f0624cba640 1 -- 172.21.15.7:0/3196326491 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061802cb50 con 0x7f062013ed70 2024-03-28T16:15:53.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.461+0000 7f0627f20640 5 monclient: authenticate success, global_id 4399 2024-03-28T16:15:53.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.461+0000 7f0627f20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:53.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.461+0000 7f0624cba640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:53.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.461+0000 7f0624cba640 1 -- 172.21.15.7:0/3196326491 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06180343f0 con 0x7f062013ed70 2024-03-28T16:15:53.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.461+0000 7f0624cba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.461+0000 7f0624cba640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:53.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.461+0000 7f0624cba640 10 monclient: dump: 2024-03-28T16:15:53.477 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.477 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.477 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.477 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.478 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.478 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.478 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.478 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.478 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.478 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.461+0000 7f0627f20640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:53.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.461+0000 7f0627f20640 4 set_mon_vals no callback set 2024-03-28T16:15:53.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.462+0000 7f0627f20640 10 monclient: shutdown 2024-03-28T16:15:53.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.462+0000 7f0627f20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:53.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.462+0000 7f0627f20640 1 -- 172.21.15.7:0/3196326491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f062013ed70 msgr2=0x7f06200b1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.462+0000 7f0627f20640 1 --2- 172.21.15.7:0/3196326491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f062013ed70 0x7f06200b1fa0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0620072060 tx=0x7f061802be50 comp rx=0 tx=0).stop 2024-03-28T16:15:53.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.462+0000 7f0627f20640 1 -- 172.21.15.7:0/3196326491 shutdown_connections 2024-03-28T16:15:53.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.462+0000 7f0627f20640 1 --2- 172.21.15.7:0/3196326491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f062013ed70 0x7f06200b1fa0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.462+0000 7f0627f20640 1 --2- 172.21.15.7:0/3196326491 >> v2:172.21.15.7:3300/0 conn(0x7f0620071920 0x7f062013e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.462+0000 7f0627f20640 1 -- 172.21.15.7:0/3196326491 >> 172.21.15.7:0/3196326491 conn(0x7f06200b3db0 msgr2=0x7f06200b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:53.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.462+0000 7f0627f20640 1 -- 172.21.15.7:0/3196326491 shutdown_connections 2024-03-28T16:15:53.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.462+0000 7f0627f20640 1 -- 172.21.15.7:0/3196326491 wait complete. 2024-03-28T16:15:53.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.463+0000 7f0627f20640 10 monclient: build_initial_monmap 2024-03-28T16:15:53.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.463+0000 7f0627f20640 10 monclient: monmap: 2024-03-28T16:15:53.480 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:53.480 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:53.480 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:53.480 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:53.480 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:53.480 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.480 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:53.480 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:53.480 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:53.480 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.463+0000 7efe12271640 1 --2- 172.21.15.7:0/3709483386 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7efe0c071920 0x7efe0c141fa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efe08033000 tx=0x7efe080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:53.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.463+0000 7efe02ffd640 10 client.?.objecter ms_handle_connect 0x7efe0c071920 2024-03-28T16:15:53.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.463+0000 7efe02ffd640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:53.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.463+0000 7f0627f20640 1 librados: starting msgr at 2024-03-28T16:15:53.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.463+0000 7f0627f20640 1 librados: starting objecter 2024-03-28T16:15:53.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.464+0000 7f0627f20640 1 Processor -- start 2024-03-28T16:15:53.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.464+0000 7f0627f20640 1 -- start start 2024-03-28T16:15:53.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.464+0000 7f0627f20640 1 librados: setting wanted keys 2024-03-28T16:15:53.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.464+0000 7f0627f20640 1 librados: calling monclient init 2024-03-28T16:15:53.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.464+0000 7f0627f20640 10 monclient: init 2024-03-28T16:15:53.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.465+0000 7f0627f20640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:53.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.465+0000 7f0627f20640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-28T16:15:53.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.465+0000 7f0627f20640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0620071920 0x7f06201375c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.465+0000 7f0627f20640 10 monclient(hunting): picked mon.noname-c con 0x7f0620071920 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:53.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.465+0000 7f0627f20640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f062013ed70 0x7f0620133000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.465+0000 7f0627f20640 10 monclient(hunting): picked mon.noname-a con 0x7f062013ed70 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.465+0000 7f0627f20640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0620133540 0x7f06201d9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.465+0000 7f0627f20640 10 monclient(hunting): picked mon.noname-b con 0x7f0620133540 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.465+0000 7f0627f20640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.465+0000 7f0627f20640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0620138790 con 0x7f062013ed70 2024-03-28T16:15:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.465+0000 7f0627f20640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.465+0000 7f0627f20640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06201339d0 con 0x7f0620133540 2024-03-28T16:15:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.465+0000 7f0627f20640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.465+0000 7f0627f20640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0620138a00 con 0x7f0620071920 2024-03-28T16:15:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.465+0000 7f0627f20640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.465+0000 7f0627f20640 10 monclient(hunting): authenticate will time out at 1420.607533s 2024-03-28T16:15:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.466+0000 7f06264bd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0620133540 0x7f06201d9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.466+0000 7f06264bd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0620133540 0x7f06201d9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:58308/0 (socket says 172.21.15.7:58308) 2024-03-28T16:15:53.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.466+0000 7f06264bd640 1 -- 172.21.15.7:0/3208556324 learned_addr learned my addr 172.21.15.7:0/3208556324 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:53.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.466+0000 7f06264bd640 10 monclient(hunting): get_auth_request con 0x7f0620133540 auth_method 0 2024-03-28T16:15:53.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.466+0000 7f06264bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:53.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.466+0000 7f06264bd640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.466+0000 7f06264bd640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:53.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.466+0000 7f06264bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:53.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.466+0000 7f06264bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:53.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.466+0000 7f06264bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:53.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.469+0000 7f06264bd640 10 monclient(hunting): handle_auth_done global_id 4405 payload 824 2024-03-28T16:15:53.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.469+0000 7f06264bd640 1 -- 172.21.15.7:0/3208556324 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0620071920 msgr2=0x7f06201375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.469+0000 7f06264bd640 1 --2- 172.21.15.7:0/3208556324 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0620071920 0x7f06201375c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.469+0000 7f06264bd640 1 -- 172.21.15.7:0/3208556324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f062013ed70 msgr2=0x7f0620133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.469+0000 7f06264bd640 1 --2- 172.21.15.7:0/3208556324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f062013ed70 0x7f0620133000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.469+0000 7f06264bd640 10 monclient: _finish_hunting 0 2024-03-28T16:15:53.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.469+0000 7f06264bd640 1 monclient: found mon.noname-b 2024-03-28T16:15:53.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.469+0000 7f06264bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:53.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.469+0000 7f06264bd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:15:53.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.469+0000 7f06264bd640 1 -- 172.21.15.7:0/3208556324 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0618009b30 con 0x7f0620133540 2024-03-28T16:15:53.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.471+0000 7f06264bd640 1 --2- 172.21.15.7:0/3208556324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0620133540 0x7f06201d9e10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0618009f40 tx=0x7f0618004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:53.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.471+0000 7f06167fc640 10 client.?.objecter ms_handle_connect 0x7f0620133540 2024-03-28T16:15:53.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.471+0000 7f06167fc640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:53.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.474+0000 7f06167fc640 1 -- 172.21.15.7:0/3208556324 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0618009510 con 0x7f0620133540 2024-03-28T16:15:53.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.474+0000 7f06167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.474+0000 7f06167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:53.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.474+0000 7f06167fc640 10 monclient: dump: 2024-03-28T16:15:53.489 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.489 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.489 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.489 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.489 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.490 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.490 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.490 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.490 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.490 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.475+0000 7f06167fc640 10 monclient: _finish_auth 0 2024-03-28T16:15:53.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.475+0000 7f06167fc640 10 monclient: _check_auth_tickets 2024-03-28T16:15:53.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.475+0000 7f06167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:53.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.475+0000 7f06167fc640 1 -- 172.21.15.7:0/3208556324 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061803e450 con 0x7f0620133540 2024-03-28T16:15:53.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.475+0000 7f0627f20640 5 monclient: authenticate success, global_id 4405 2024-03-28T16:15:53.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.475+0000 7f0627f20640 10 monclient: _renew_subs 2024-03-28T16:15:53.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.475+0000 7f0627f20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:53.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.475+0000 7f0627f20640 1 -- 172.21.15.7:0/3208556324 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0620138ce0 con 0x7f0620133540 2024-03-28T16:15:53.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.475+0000 7f06167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:53.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.475+0000 7f06167fc640 1 -- 172.21.15.7:0/3208556324 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061803e800 con 0x7f0620133540 2024-03-28T16:15:53.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.475+0000 7f06167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.475+0000 7f06167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:53.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.475+0000 7f0627f20640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:53.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.475+0000 7f06167fc640 10 monclient: dump: 2024-03-28T16:15:53.491 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.491 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.491 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.491 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.491 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.491 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.491 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.491 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.491 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.491 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.475+0000 7f0617fff640 4 set_mon_vals no callback set 2024-03-28T16:15:53.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.475+0000 7f0627f20640 10 monclient: _renew_subs 2024-03-28T16:15:53.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.475+0000 7f0627f20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:53.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.475+0000 7f0627f20640 1 -- 172.21.15.7:0/3208556324 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0620144450 con 0x7f0620133540 2024-03-28T16:15:53.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.475+0000 7f0627f20640 1 librados: init done 2024-03-28T16:15:53.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.476+0000 7f0627f20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:53.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.476+0000 7f0627f20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:53.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.476+0000 7f0627f20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:53.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.476+0000 7f0627f20640 1 -- 172.21.15.7:0/3208556324 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0620069430 con 0x7f0620133540 2024-03-28T16:15:53.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.494+0000 7f06167fc640 1 -- 172.21.15.7:0/3208556324 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f061803e960 con 0x7f0620133540 2024-03-28T16:15:53.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.494+0000 7f06167fc640 1 --2- 172.21.15.7:0/3208556324 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f060406c590 0x7f060406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.495+0000 7f0625cbc640 1 --2- 172.21.15.7:0/3208556324 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f060406c590 0x7f060406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.495+0000 7f0625cbc640 10 monclient: get_auth_request con 0x7f060406c590 auth_method 0 2024-03-28T16:15:53.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.495+0000 7f0625cbc640 1 --2- 172.21.15.7:0/3208556324 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f060406c590 0x7f060406ea50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f061c009710 tx=0x7f061c006c90 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:53.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.495+0000 7f06167fc640 10 client.4405.objecter ms_handle_connect 0x7f060406c590 2024-03-28T16:15:53.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.496+0000 7f06167fc640 1 -- 172.21.15.7:0/3208556324 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f061807bc10 con 0x7f0620133540 2024-03-28T16:15:53.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.496+0000 7f06167fc640 10 client.4405.objecter ms_dispatch 0x7f0620000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:53.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.496+0000 7f06167fc640 3 client.4405.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:53.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.496+0000 7f06167fc640 3 client.4405.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:53.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.496+0000 7f06167fc640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-28T16:15:53.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.511+0000 7fb70ffff640 1 -- 172.21.15.7:0/446609966 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb71801a400 con 0x7fb728071480 2024-03-28T16:15:53.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.511+0000 7fb70ffff640 1 --2- 172.21.15.7:0/446609966 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fb71406c5e0 0x7fb71406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.512+0000 7fb7267fc640 1 --2- 172.21.15.7:0/446609966 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fb71406c5e0 0x7fb71406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.512+0000 7fb7267fc640 10 monclient: get_auth_request con 0x7fb71406c5e0 auth_method 0 2024-03-28T16:15:53.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.512+0000 7fb7267fc640 1 --2- 172.21.15.7:0/446609966 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fb71406c5e0 0x7fb71406eaa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb71c035000 tx=0x7fb71c030000 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:53.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.512+0000 7fb70ffff640 10 client.4304.objecter ms_handle_connect 0x7fb71406c5e0 2024-03-28T16:15:53.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.535+0000 7f23977fe640 1 -- 172.21.15.7:0/605213623 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f23a402f050 con 0x7f23b00b1890 2024-03-28T16:15:53.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.535+0000 7f23977fe640 1 --2- 172.21.15.7:0/605213623 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f239c06c610 0x7f239c06ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.536+0000 7f23aed9d640 1 --2- 172.21.15.7:0/605213623 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f239c06c610 0x7f239c06ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.536+0000 7f23aed9d640 10 monclient: get_auth_request con 0x7f239c06c610 auth_method 0 2024-03-28T16:15:53.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.536+0000 7f23977fe640 10 client.4353.objecter ms_handle_connect 0x7f239c06c610 2024-03-28T16:15:53.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.536+0000 7f23aed9d640 1 --2- 172.21.15.7:0/605213623 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f239c06c610 0x7f239c06ead0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f23a0009710 tx=0x7f23a0006c90 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:53.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.557+0000 7f23977fe640 1 -- 172.21.15.7:0/605213623 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f23a407ada0 con 0x7f23b00b1890 2024-03-28T16:15:53.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.557+0000 7f23977fe640 10 client.4353.objecter ms_dispatch 0x7f23b0000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:53.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.557+0000 7f23977fe640 3 client.4353.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:53.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.557+0000 7f23977fe640 3 client.4353.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:53.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.557+0000 7f23977fe640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-28T16:15:53.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.557+0000 7f23977fe640 1 -- 172.21.15.7:0/605213623 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f23a40769e0 con 0x7f23b00b1890 2024-03-28T16:15:53.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.557+0000 7f23977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:53.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.557+0000 7f23977fe640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:53.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.585+0000 7f22377fe640 1 --2- 172.21.15.7:0/1417382972 >> v2:172.21.15.7:3300/0 conn(0x7f22380a86e0 0x7f2238145110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.590+0000 7f22377fe640 10 monclient(hunting): get_auth_request con 0x7f22380a86e0 auth_method 0 2024-03-28T16:15:53.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.590+0000 7f22377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:53.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.590+0000 7f22377fe640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.590+0000 7f22377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:53.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.592+0000 7f22377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:53.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.592+0000 7f22377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:53.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.592+0000 7f22377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:53.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.595+0000 7f22377fe640 10 monclient(hunting): handle_auth_done global_id 4356 payload 274 2024-03-28T16:15:53.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.595+0000 7f22377fe640 1 -- 172.21.15.7:0/1417382972 >> v1:172.21.15.83:6789/0 conn(0x7f22380b32e0 legacy=0x7f2238144a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.595+0000 7f22377fe640 1 -- 172.21.15.7:0/1417382972 >> v1:172.21.15.7:6789/0 conn(0x7f2238071480 legacy=0x7f223813e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.595+0000 7f22377fe640 10 monclient: _finish_hunting 0 2024-03-28T16:15:53.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.595+0000 7f22377fe640 1 monclient: found mon.noname-a 2024-03-28T16:15:53.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.595+0000 7f22377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:53.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.595+0000 7f22377fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:15:53.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.595+0000 7f22377fe640 1 -- 172.21.15.7:0/1417382972 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223813f020 con 0x7f22380a86e0 2024-03-28T16:15:53.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.600+0000 7f22377fe640 1 --2- 172.21.15.7:0/1417382972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22380a86e0 0x7f2238145110 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2230009d90 tx=0x7f223002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b438d9092f88cf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:53.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.633+0000 7fd3344c4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-28T16:15:53.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.633+0000 7fd3344c4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-28T16:15:53.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.633+0000 7fd3344c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:53.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.633+0000 7fd3344c4640 1 -- 172.21.15.7:0/3651179925 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fd32c139ec0 con 0x7fd32c0a86e0 2024-03-28T16:15:53.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.636+0000 7f8ed788e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-28T16:15:53.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.636+0000 7f8ed788e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-28T16:15:53.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.636+0000 7f8ed788e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:53.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.636+0000 7f8ed788e640 1 -- 172.21.15.7:0/4243004799 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f8ed0139a80 con 0x7f8ed011be70 2024-03-28T16:15:53.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.702+0000 7f2235ffb640 1 -- 172.21.15.7:0/1417382972 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223002cbd0 con 0x7f22380a86e0 2024-03-28T16:15:53.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.702+0000 7f2235ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.702+0000 7f2235ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:15:53.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.702+0000 7f2235ffb640 10 monclient: dump: 2024-03-28T16:15:53.703 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.703 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.704 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.704 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.704 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.704 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.704 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.704 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.704 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.704 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.702+0000 7f2235ffb640 10 monclient: _finish_auth 0 2024-03-28T16:15:53.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.702+0000 7f2235ffb640 10 monclient: _check_auth_tickets 2024-03-28T16:15:53.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.702+0000 7f2235ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:53.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.702+0000 7f2235ffb640 1 -- 172.21.15.7:0/1417382972 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f223002cd30 con 0x7f22380a86e0 2024-03-28T16:15:53.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.702+0000 7f223d7e4640 5 monclient: authenticate success, global_id 4356 2024-03-28T16:15:53.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.702+0000 7f223d7e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:53.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.702+0000 7f2235ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:53.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.702+0000 7f223d7e4640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:53.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.702+0000 7f2235ffb640 1 -- 172.21.15.7:0/1417382972 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223002cea0 con 0x7f22380a86e0 2024-03-28T16:15:53.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.702+0000 7f2235ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.702+0000 7f223d7e4640 4 set_mon_vals no callback set 2024-03-28T16:15:53.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.702+0000 7f2235ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:15:53.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.702+0000 7f2235ffb640 10 monclient: dump: 2024-03-28T16:15:53.705 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.705 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.705 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.705 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.705 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.705 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.706 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.706 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.706 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.706 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.703+0000 7f223d7e4640 10 monclient: shutdown 2024-03-28T16:15:53.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.703+0000 7f223d7e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:53.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.703+0000 7f223d7e4640 1 -- 172.21.15.7:0/1417382972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22380a86e0 msgr2=0x7f2238145110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.703+0000 7f223d7e4640 1 --2- 172.21.15.7:0/1417382972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22380a86e0 0x7f2238145110 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2230009d90 tx=0x7f223002be20 comp rx=0 tx=0).stop 2024-03-28T16:15:53.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.703+0000 7f223d7e4640 1 -- 172.21.15.7:0/1417382972 shutdown_connections 2024-03-28T16:15:53.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.703+0000 7f223d7e4640 1 --2- 172.21.15.7:0/1417382972 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22380a86e0 0x7f2238145110 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.703+0000 7f223d7e4640 1 -- 172.21.15.7:0/1417382972 >> 172.21.15.7:0/1417382972 conn(0x7f22380b70e0 msgr2=0x7f22380b74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:53.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.703+0000 7f223d7e4640 1 -- 172.21.15.7:0/1417382972 shutdown_connections 2024-03-28T16:15:53.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.703+0000 7f223d7e4640 1 -- 172.21.15.7:0/1417382972 wait complete. 2024-03-28T16:15:53.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.704+0000 7f223d7e4640 10 monclient: build_initial_monmap 2024-03-28T16:15:53.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.704+0000 7f223d7e4640 10 monclient: monmap: 2024-03-28T16:15:53.709 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:53.709 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:53.709 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:53.709 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:53.709 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:53.709 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.709 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:53.709 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:53.710 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:53.710 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.704+0000 7f223d7e4640 1 librados: starting msgr at 2024-03-28T16:15:53.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.704+0000 7f223d7e4640 1 librados: starting objecter 2024-03-28T16:15:53.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.705+0000 7f223d7e4640 1 Processor -- start 2024-03-28T16:15:53.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.705+0000 7f223d7e4640 1 -- start start 2024-03-28T16:15:53.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.705+0000 7f223d7e4640 1 librados: setting wanted keys 2024-03-28T16:15:53.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.705+0000 7f223d7e4640 1 librados: calling monclient init 2024-03-28T16:15:53.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.705+0000 7f223d7e4640 10 monclient: init 2024-03-28T16:15:53.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.705+0000 7f223d7e4640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:53.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.705+0000 7f223d7e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-28T16:15:53.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.705+0000 7f223d7e4640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2238071480 0x7f223813d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.706+0000 7f223d7e4640 10 monclient(hunting): picked mon.noname-c con 0x7f2238071480 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:53.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.706+0000 7f223d7e4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22380a86e0 0x7f2238140770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.706+0000 7f223d7e4640 10 monclient(hunting): picked mon.noname-b con 0x7f22380a86e0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:53.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.706+0000 7f223d7e4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22380b32e0 0x7f2238143b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.706+0000 7f223d7e4640 10 monclient(hunting): picked mon.noname-a con 0x7f22380b32e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:53.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.706+0000 7f223d7e4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.706+0000 7f223d7e4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2238142610 con 0x7f22380b32e0 2024-03-28T16:15:53.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.706+0000 7f223d7e4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.706+0000 7f223d7e4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223813e0f0 con 0x7f22380a86e0 2024-03-28T16:15:53.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.706+0000 7f223d7e4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:53.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.706+0000 7f223d7e4640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2238140cb0 con 0x7f2238071480 2024-03-28T16:15:53.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.706+0000 7f223d7e4640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:53.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.706+0000 7f223d7e4640 10 monclient(hunting): authenticate will time out at 1420.847908s 2024-03-28T16:15:53.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.706+0000 7f22377fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22380b32e0 0x7f2238143b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.707+0000 7f22377fe640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22380b32e0 0x7f2238143b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38134/0 (socket says 172.21.15.7:38134) 2024-03-28T16:15:53.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.707+0000 7f22377fe640 1 -- 172.21.15.7:0/1499334056 learned_addr learned my addr 172.21.15.7:0/1499334056 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:53.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.707+0000 7f22377fe640 10 monclient(hunting): get_auth_request con 0x7f22380b32e0 auth_method 0 2024-03-28T16:15:53.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.707+0000 7f22377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:53.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.707+0000 7f22377fe640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:53.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.707+0000 7f22377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:53.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.707+0000 7f22377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:53.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.707+0000 7f22377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:53.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.707+0000 7f22377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:53.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.710+0000 7f22377fe640 10 monclient(hunting): handle_auth_done global_id 4359 payload 824 2024-03-28T16:15:53.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.710+0000 7f22377fe640 1 -- 172.21.15.7:0/1499334056 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2238071480 msgr2=0x7f223813d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.710+0000 7f22377fe640 1 --2- 172.21.15.7:0/1499334056 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2238071480 0x7f223813d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.710+0000 7f22377fe640 1 -- 172.21.15.7:0/1499334056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22380a86e0 msgr2=0x7f2238140770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.710+0000 7f22377fe640 1 --2- 172.21.15.7:0/1499334056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22380a86e0 0x7f2238140770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.710+0000 7f22377fe640 10 monclient: _finish_hunting 0 2024-03-28T16:15:53.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.710+0000 7f22377fe640 1 monclient: found mon.noname-a 2024-03-28T16:15:53.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.710+0000 7f22377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:53.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.710+0000 7f22377fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:15:53.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.710+0000 7f22377fe640 1 -- 172.21.15.7:0/1499334056 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2230009a70 con 0x7f22380b32e0 2024-03-28T16:15:53.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.712+0000 7f22377fe640 1 --2- 172.21.15.7:0/1499334056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22380b32e0 0x7f2238143b40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2230037040 tx=0x7f2230004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:53.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.712+0000 7f221f7fe640 10 client.?.objecter ms_handle_connect 0x7f22380b32e0 2024-03-28T16:15:53.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.712+0000 7f221f7fe640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:53.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f221f7fe640 1 -- 172.21.15.7:0/1499334056 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223002cea0 con 0x7f22380b32e0 2024-03-28T16:15:53.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f221f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f221f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:15:53.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f221f7fe640 10 monclient: dump: 2024-03-28T16:15:53.718 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.718 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.718 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.718 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.718 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.718 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.718 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.718 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.718 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.718 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f221f7fe640 10 monclient: _finish_auth 0 2024-03-28T16:15:53.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f221f7fe640 10 monclient: _check_auth_tickets 2024-03-28T16:15:53.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f221f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:53.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f221f7fe640 1 -- 172.21.15.7:0/1499334056 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2230040650 con 0x7f22380b32e0 2024-03-28T16:15:53.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f221f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:53.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f221f7fe640 1 -- 172.21.15.7:0/1499334056 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2230040a00 con 0x7f22380b32e0 2024-03-28T16:15:53.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f221f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f221f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:15:53.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f221f7fe640 10 monclient: dump: 2024-03-28T16:15:53.719 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.719 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.719 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.719 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.719 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.719 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.720 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.720 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.720 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.720 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f223d7e4640 5 monclient: authenticate success, global_id 4359 2024-03-28T16:15:53.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f223d7e4640 10 monclient: _renew_subs 2024-03-28T16:15:53.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f223d7e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:53.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f223d7e4640 1 -- 172.21.15.7:0/1499334056 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2238140070 con 0x7f22380b32e0 2024-03-28T16:15:53.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f22357fa640 4 set_mon_vals no callback set 2024-03-28T16:15:53.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f223d7e4640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:53.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f223d7e4640 10 monclient: _renew_subs 2024-03-28T16:15:53.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f223d7e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:53.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f223d7e4640 1 -- 172.21.15.7:0/1499334056 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22381446a0 con 0x7f22380b32e0 2024-03-28T16:15:53.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.716+0000 7f223d7e4640 1 librados: init done 2024-03-28T16:15:53.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.717+0000 7f223d7e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:53.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.717+0000 7f223d7e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:53.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.717+0000 7f223d7e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:53.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.717+0000 7f223d7e4640 1 -- 172.21.15.7:0/1499334056 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f223813df30 con 0x7f22380b32e0 2024-03-28T16:15:53.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.733+0000 7f06167fc640 1 -- 172.21.15.7:0/3208556324 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0618077850 con 0x7f0620133540 2024-03-28T16:15:53.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.733+0000 7f06167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:53.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.733+0000 7f06167fc640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:53.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.751+0000 7f221f7fe640 1 -- 172.21.15.7:0/1499334056 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2230040b60 con 0x7f22380b32e0 2024-03-28T16:15:53.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.751+0000 7f221f7fe640 1 --2- 172.21.15.7:0/1499334056 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f222406c590 0x7f222406ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.751+0000 7f2236ffd640 1 --2- 172.21.15.7:0/1499334056 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f222406c590 0x7f222406ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.751+0000 7f2236ffd640 10 monclient: get_auth_request con 0x7f222406c590 auth_method 0 2024-03-28T16:15:53.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.752+0000 7f221f7fe640 10 client.4359.objecter ms_handle_connect 0x7f222406c590 2024-03-28T16:15:53.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.752+0000 7f2236ffd640 1 --2- 172.21.15.7:0/1499334056 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f222406c590 0x7f222406ea50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2228000f60 tx=0x7f22280079b0 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:53.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.763+0000 7f23b51a4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-28T16:15:53.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.763+0000 7f23b51a4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-28T16:15:53.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.763+0000 7f23b51a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:53.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.763+0000 7f23b51a4640 1 -- 172.21.15.7:0/605213623 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f23b0144d00 con 0x7f23b00b1890 2024-03-28T16:15:53.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.793+0000 7fb70ffff640 1 -- 172.21.15.7:0/446609966 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb7180591b0 con 0x7fb728071480 2024-03-28T16:15:53.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.793+0000 7fb70ffff640 10 client.4304.objecter ms_dispatch 0x7fb728000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:53.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.793+0000 7fb70ffff640 3 client.4304.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:53.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.793+0000 7fb70ffff640 3 client.4304.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:53.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.793+0000 7fb70ffff640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-28T16:15:53.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.794+0000 7fb70ffff640 1 -- 172.21.15.7:0/446609966 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb718054df0 con 0x7fb728071480 2024-03-28T16:15:53.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.794+0000 7fb70ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:53.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.794+0000 7fb70ffff640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:53.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.808+0000 7f8ec5ffb640 1 -- 172.21.15.7:0/4243004799 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8ec007ab10 con 0x7f8ed011be70 2024-03-28T16:15:53.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.808+0000 7f8ec5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-28T16:15:53.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.808+0000 7f8ec5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:53.809 INFO:teuthology.orchestra.run.smithi007.stdout:73014444038 2024-03-28T16:15:53.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.810+0000 7f8ed788e640 10 librados: watch_flush enter 2024-03-28T16:15:53.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.810+0000 7f8ed788e640 10 librados: watch_flush exit 2024-03-28T16:15:53.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.810+0000 7f8ed788e640 20 client.4387.objecter shutdown clearing up homeless session... 2024-03-28T16:15:53.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.810+0000 7f8ed788e640 10 client.4387.objecter successfully canceled tick 2024-03-28T16:15:53.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.810+0000 7f8ed788e640 1 -- 172.21.15.7:0/4243004799 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f8eb406c590 msgr2=0x7f8eb406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.810+0000 7f8ed788e640 1 --2- 172.21.15.7:0/4243004799 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f8eb406c590 0x7f8eb406ea50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc0011e0 tx=0x7f8ecc0014f0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.810+0000 7f8ed788e640 10 monclient: shutdown 2024-03-28T16:15:53.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.810+0000 7f8ed788e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:53.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.810+0000 7f8ed788e640 1 -- 172.21.15.7:0/4243004799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ed011be70 msgr2=0x7f8ed0137e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.810+0000 7f8ed788e640 1 --2- 172.21.15.7:0/4243004799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ed011be70 0x7f8ed0137e20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8ec002bd70 tx=0x7f8ec00012d0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.811+0000 7f8ed788e640 1 -- 172.21.15.7:0/4243004799 shutdown_connections 2024-03-28T16:15:53.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.811+0000 7f8ed788e640 1 --2- 172.21.15.7:0/4243004799 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f8eb406c590 0x7f8eb406ea50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.811+0000 7f8ed788e640 1 --2- 172.21.15.7:0/4243004799 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8ed0138360 0x7f8ed01d6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.811+0000 7f8ed788e640 1 --2- 172.21.15.7:0/4243004799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ed011be70 0x7f8ed0137e20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.811+0000 7f8ed788e640 1 --2- 172.21.15.7:0/4243004799 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8ebc0042a0 0x7f8ed011cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.811+0000 7f8ed788e640 1 -- 172.21.15.7:0/4243004799 >> 172.21.15.7:0/4243004799 conn(0x7f8ed00b3dc0 msgr2=0x7f8ed00ae4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:53.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.811+0000 7f8ed788e640 1 -- 172.21.15.7:0/4243004799 shutdown_connections 2024-03-28T16:15:53.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.811+0000 7f8ed788e640 1 -- 172.21.15.7:0/4243004799 wait complete. 2024-03-28T16:15:53.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.811+0000 7f8ed788e640 1 librados: shutdown 2024-03-28T16:15:53.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.825+0000 7fd322ffd640 1 -- 172.21.15.7:0/3651179925 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd31c038020 con 0x7fd32c0a86e0 2024-03-28T16:15:53.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.825+0000 7fd322ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-28T16:15:53.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.825+0000 7fd322ffd640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:53.827 INFO:teuthology.orchestra.run.smithi007.stdout:64424509448 2024-03-28T16:15:53.828 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444038 got 73014444038 for osd.1 2024-03-28T16:15:53.828 DEBUG:teuthology.parallel:result is None 2024-03-28T16:15:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.827+0000 7fd3344c4640 10 librados: watch_flush enter 2024-03-28T16:15:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.827+0000 7fd3344c4640 10 librados: watch_flush exit 2024-03-28T16:15:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.827+0000 7fd3344c4640 20 client.4301.objecter shutdown clearing up homeless session... 2024-03-28T16:15:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.827+0000 7fd3344c4640 10 client.4301.objecter successfully canceled tick 2024-03-28T16:15:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.827+0000 7fd3344c4640 1 -- 172.21.15.7:0/3651179925 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fd31006c590 msgr2=0x7fd31006ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.827+0000 7fd3344c4640 1 --2- 172.21.15.7:0/3651179925 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fd31006c590 0x7fd31006ea50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd328006fd0 tx=0x7fd328009060 comp rx=0 tx=0).stop 2024-03-28T16:15:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.827+0000 7fd3344c4640 10 monclient: shutdown 2024-03-28T16:15:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.827+0000 7fd3344c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.827+0000 7fd3344c4640 1 -- 172.21.15.7:0/3651179925 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd32c0a86e0 msgr2=0x7fd32c143790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.827+0000 7fd3344c4640 1 --2- 172.21.15.7:0/3651179925 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd32c0a86e0 0x7fd32c143790 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd31c000c00 tx=0x7fd31c0046c0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.827+0000 7fd3344c4640 1 -- 172.21.15.7:0/3651179925 shutdown_connections 2024-03-28T16:15:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.827+0000 7fd3344c4640 1 --2- 172.21.15.7:0/3651179925 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fd31006c590 0x7fd31006ea50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.827+0000 7fd3344c4640 1 --2- 172.21.15.7:0/3651179925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd32c0b32e0 0x7fd32c143cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.827+0000 7fd3344c4640 1 --2- 172.21.15.7:0/3651179925 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd32c0a86e0 0x7fd32c143790 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.827+0000 7fd3344c4640 1 --2- 172.21.15.7:0/3651179925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd32c071480 0x7fd32c1381b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.827+0000 7fd3344c4640 1 -- 172.21.15.7:0/3651179925 >> 172.21.15.7:0/3651179925 conn(0x7fd32c0b70e0 msgr2=0x7fd32c0b4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.827+0000 7fd3344c4640 1 -- 172.21.15.7:0/3651179925 shutdown_connections 2024-03-28T16:15:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.827+0000 7fd3344c4640 1 -- 172.21.15.7:0/3651179925 wait complete. 2024-03-28T16:15:53.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.827+0000 7fd3344c4640 1 librados: shutdown 2024-03-28T16:15:53.840 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509448 for osd.7 2024-03-28T16:15:53.840 DEBUG:teuthology.parallel:result is None 2024-03-28T16:15:53.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.925+0000 7efe02ffd640 1 -- 172.21.15.7:0/3709483386 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe08032940 con 0x7efe0c071920 2024-03-28T16:15:53.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.925+0000 7efe02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.925+0000 7efe02ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:53.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.925+0000 7efe02ffd640 10 monclient: dump: 2024-03-28T16:15:53.927 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.928 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.928 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.928 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.928 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.928 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.928 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.928 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.928 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.928 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.925+0000 7efe02ffd640 10 monclient: _finish_auth 0 2024-03-28T16:15:53.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.925+0000 7efe02ffd640 10 monclient: _check_auth_tickets 2024-03-28T16:15:53.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.925+0000 7efe02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:53.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.926+0000 7efe02ffd640 1 -- 172.21.15.7:0/3709483386 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe08032aa0 con 0x7efe0c071920 2024-03-28T16:15:53.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.926+0000 7efe144d5640 5 monclient: authenticate success, global_id 4307 2024-03-28T16:15:53.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.926+0000 7efe144d5640 10 monclient: _renew_subs 2024-03-28T16:15:53.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.926+0000 7efe144d5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:53.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.926+0000 7efe144d5640 1 -- 172.21.15.7:0/3709483386 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe0c0b68a0 con 0x7efe0c071920 2024-03-28T16:15:53.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.926+0000 7efe02ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:53.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.926+0000 7efe02ffd640 1 -- 172.21.15.7:0/3709483386 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe08032e50 con 0x7efe0c071920 2024-03-28T16:15:53.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.926+0000 7efe02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:53.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.926+0000 7efe02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:15:53.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.926+0000 7efe02ffd640 10 monclient: dump: 2024-03-28T16:15:53.929 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:53.929 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:53.929 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.929 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:53.929 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:53.929 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:53.929 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:53.930 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:53.930 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:53.930 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:53.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.926+0000 7efe144d5640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:53.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.926+0000 7efe144d5640 10 monclient: _renew_subs 2024-03-28T16:15:53.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.926+0000 7efe144d5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:53.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.926+0000 7efe03fff640 4 set_mon_vals no callback set 2024-03-28T16:15:53.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.926+0000 7efe144d5640 1 -- 172.21.15.7:0/3709483386 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe0c145890 con 0x7efe0c071920 2024-03-28T16:15:53.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.926+0000 7efe144d5640 1 librados: init done 2024-03-28T16:15:53.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.927+0000 7efe144d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:53.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.927+0000 7efe144d5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:53.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.927+0000 7efe144d5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:53.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.927+0000 7efe144d5640 1 -- 172.21.15.7:0/3709483386 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe0c069430 con 0x7efe0c071920 2024-03-28T16:15:53.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.934+0000 7f0627f20640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-28T16:15:53.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.934+0000 7f0627f20640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-28T16:15:53.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.934+0000 7f0627f20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:53.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.934+0000 7f0627f20640 1 -- 172.21.15.7:0/3208556324 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f06200aa6c0 con 0x7f0620133540 2024-03-28T16:15:53.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.937+0000 7f06167fc640 1 -- 172.21.15.7:0/3208556324 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0618043090 con 0x7f0620133540 2024-03-28T16:15:53.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.937+0000 7f06167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-28T16:15:53.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.937+0000 7f06167fc640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:53.939 INFO:teuthology.orchestra.run.smithi007.stdout:64424509446 2024-03-28T16:15:53.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.940+0000 7f0627f20640 10 librados: watch_flush enter 2024-03-28T16:15:53.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.940+0000 7f0627f20640 10 librados: watch_flush exit 2024-03-28T16:15:53.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.940+0000 7f0627f20640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-28T16:15:53.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.940+0000 7f0627f20640 10 client.4405.objecter successfully canceled tick 2024-03-28T16:15:53.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.940+0000 7f0627f20640 1 -- 172.21.15.7:0/3208556324 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f060406c590 msgr2=0x7f060406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.940+0000 7f0627f20640 1 --2- 172.21.15.7:0/3208556324 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f060406c590 0x7f060406ea50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f061c009710 tx=0x7f061c006c90 comp rx=0 tx=0).stop 2024-03-28T16:15:53.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.940+0000 7f0627f20640 10 monclient: shutdown 2024-03-28T16:15:53.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.940+0000 7f0627f20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:53.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.940+0000 7f0627f20640 1 -- 172.21.15.7:0/3208556324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0620133540 msgr2=0x7f06201d9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:53.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.940+0000 7f0627f20640 1 --2- 172.21.15.7:0/3208556324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0620133540 0x7f06201d9e10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0618009f40 tx=0x7f0618004a40 comp rx=0 tx=0).stop 2024-03-28T16:15:53.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.941+0000 7f0627f20640 1 -- 172.21.15.7:0/3208556324 shutdown_connections 2024-03-28T16:15:53.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.941+0000 7f0627f20640 1 --2- 172.21.15.7:0/3208556324 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f060406c590 0x7f060406ea50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.941+0000 7f0627f20640 1 --2- 172.21.15.7:0/3208556324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f062013ed70 0x7f0620133000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.941+0000 7f0627f20640 1 --2- 172.21.15.7:0/3208556324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0620133540 0x7f06201d9e10 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.941+0000 7f0627f20640 1 --2- 172.21.15.7:0/3208556324 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0620071920 0x7f06201375c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:53.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.941+0000 7f0627f20640 1 -- 172.21.15.7:0/3208556324 >> 172.21.15.7:0/3208556324 conn(0x7f06200b3db0 msgr2=0x7f062013b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:53.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.941+0000 7f0627f20640 1 -- 172.21.15.7:0/3208556324 shutdown_connections 2024-03-28T16:15:53.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.941+0000 7f0627f20640 1 -- 172.21.15.7:0/3208556324 wait complete. 2024-03-28T16:15:53.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.941+0000 7f0627f20640 1 librados: shutdown 2024-03-28T16:15:53.956 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.0 2024-03-28T16:15:53.956 DEBUG:teuthology.parallel:result is None 2024-03-28T16:15:53.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.957+0000 7efe02ffd640 1 -- 172.21.15.7:0/3709483386 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7efe0804e450 con 0x7efe0c071920 2024-03-28T16:15:53.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.957+0000 7efe02ffd640 1 --2- 172.21.15.7:0/3709483386 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7efdf006c5e0 0x7efdf006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:53.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.957+0000 7efe11a70640 1 --2- 172.21.15.7:0/3709483386 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7efdf006c5e0 0x7efdf006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:53.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.957+0000 7efe11a70640 10 monclient: get_auth_request con 0x7efdf006c5e0 auth_method 0 2024-03-28T16:15:53.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.958+0000 7efe11a70640 1 --2- 172.21.15.7:0/3709483386 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7efdf006c5e0 0x7efdf006eaa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efdfc007fd0 tx=0x7efdfc007980 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:53.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:53.958+0000 7efe02ffd640 10 client.4307.objecter ms_handle_connect 0x7efdf006c5e0 2024-03-28T16:15:54.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.028+0000 7f221f7fe640 1 -- 172.21.15.7:0/1499334056 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f223007bbb0 con 0x7f22380b32e0 2024-03-28T16:15:54.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.028+0000 7f221f7fe640 10 client.4359.objecter ms_dispatch 0x7f2238000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:54.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.028+0000 7f221f7fe640 3 client.4359.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:54.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.028+0000 7f221f7fe640 3 client.4359.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:54.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.028+0000 7f221f7fe640 20 client.4359.objecter dump_active .. 0 homeless 2024-03-28T16:15:54.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.028+0000 7f221f7fe640 1 -- 172.21.15.7:0/1499334056 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f22300777f0 con 0x7f22380b32e0 2024-03-28T16:15:54.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.028+0000 7f221f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:54.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.028+0000 7f221f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:54.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.035+0000 7f23977fe640 1 -- 172.21.15.7:0/605213623 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f23a4031e60 con 0x7f23b00b1890 2024-03-28T16:15:54.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.035+0000 7f23977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-28T16:15:54.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.035+0000 7f23977fe640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:54.036 INFO:teuthology.orchestra.run.smithi007.stdout:98784247816 2024-03-28T16:15:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.037+0000 7f23b51a4640 10 librados: watch_flush enter 2024-03-28T16:15:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.037+0000 7f23b51a4640 10 librados: watch_flush exit 2024-03-28T16:15:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.037+0000 7f23b51a4640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-28T16:15:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.037+0000 7f23b51a4640 10 client.4353.objecter successfully canceled tick 2024-03-28T16:15:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.037+0000 7f23b51a4640 1 -- 172.21.15.7:0/605213623 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f239c06c610 msgr2=0x7f239c06ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.037+0000 7f23b51a4640 1 --2- 172.21.15.7:0/605213623 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f239c06c610 0x7f239c06ead0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f23a0009710 tx=0x7f23a0006c90 comp rx=0 tx=0).stop 2024-03-28T16:15:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.037+0000 7f23b51a4640 10 monclient: shutdown 2024-03-28T16:15:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.037+0000 7f23b51a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.037+0000 7f23b51a4640 1 -- 172.21.15.7:0/605213623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23b00b1890 msgr2=0x7f23b01443d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:54.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.037+0000 7f23b51a4640 1 --2- 172.21.15.7:0/605213623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23b00b1890 0x7f23b01443d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f23a4009870 tx=0x7f23a40046c0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.037+0000 7f23b51a4640 1 -- 172.21.15.7:0/605213623 shutdown_connections 2024-03-28T16:15:54.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.037+0000 7f23b51a4640 1 --2- 172.21.15.7:0/605213623 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f239c06c610 0x7f239c06ead0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.037+0000 7f23b51a4640 1 --2- 172.21.15.7:0/605213623 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23b00b1890 0x7f23b01443d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.037+0000 7f23b51a4640 1 --2- 172.21.15.7:0/605213623 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23b0140920 0x7f23b0145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.037+0000 7f23b51a4640 1 --2- 172.21.15.7:0/605213623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23b0071920 0x7f23b00b61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.038+0000 7f23b51a4640 1 -- 172.21.15.7:0/605213623 >> 172.21.15.7:0/605213623 conn(0x7f23b00b3db0 msgr2=0x7f23b013e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:54.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.038+0000 7f23b51a4640 1 -- 172.21.15.7:0/605213623 shutdown_connections 2024-03-28T16:15:54.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.038+0000 7f23b51a4640 1 -- 172.21.15.7:0/605213623 wait complete. 2024-03-28T16:15:54.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.038+0000 7f23b51a4640 1 librados: shutdown 2024-03-28T16:15:54.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.046+0000 7fb72d9ab640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-28T16:15:54.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.046+0000 7fb72d9ab640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-28T16:15:54.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.046+0000 7fb72d9ab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:54.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.046+0000 7fb72d9ab640 1 -- 172.21.15.7:0/446609966 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fb728139e10 con 0x7fb728071480 2024-03-28T16:15:54.050 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247815 got 98784247816 for osd.5 2024-03-28T16:15:54.051 DEBUG:teuthology.parallel:result is None 2024-03-28T16:15:54.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.221+0000 7efe02ffd640 1 -- 172.21.15.7:0/3709483386 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7efe0807af60 con 0x7efe0c071920 2024-03-28T16:15:54.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.221+0000 7efe02ffd640 10 client.4307.objecter ms_dispatch 0x7efe0c000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:54.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.221+0000 7efe02ffd640 3 client.4307.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:54.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.221+0000 7efe02ffd640 3 client.4307.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:54.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.222+0000 7efe02ffd640 20 client.4307.objecter dump_active .. 0 homeless 2024-03-28T16:15:54.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.222+0000 7efe02ffd640 1 -- 172.21.15.7:0/3709483386 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efe08076ba0 con 0x7efe0c071920 2024-03-28T16:15:54.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.222+0000 7efe02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:54.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.222+0000 7efe02ffd640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:54.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.230+0000 7fb70ffff640 1 -- 172.21.15.7:0/446609966 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb718015020 con 0x7fb728071480 2024-03-28T16:15:54.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.230+0000 7fb70ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-28T16:15:54.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.230+0000 7fb70ffff640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:54.232 INFO:teuthology.orchestra.run.smithi007.stdout:60129542150 2024-03-28T16:15:54.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.232+0000 7fb72d9ab640 10 librados: watch_flush enter 2024-03-28T16:15:54.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.232+0000 7fb72d9ab640 10 librados: watch_flush exit 2024-03-28T16:15:54.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.232+0000 7fb72d9ab640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-28T16:15:54.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.232+0000 7fb72d9ab640 10 client.4304.objecter successfully canceled tick 2024-03-28T16:15:54.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.232+0000 7f223d7e4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-28T16:15:54.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.232+0000 7f223d7e4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-28T16:15:54.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.232+0000 7f223d7e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2024-03-28T16:15:54.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.232+0000 7f223d7e4640 1 -- 172.21.15.7:0/1499334056 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f223813e2b0 con 0x7f22380b32e0 2024-03-28T16:15:54.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.232+0000 7fb72d9ab640 1 -- 172.21.15.7:0/446609966 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fb71406c5e0 msgr2=0x7fb71406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:54.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.232+0000 7fb72d9ab640 1 --2- 172.21.15.7:0/446609966 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fb71406c5e0 0x7fb71406eaa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb71c035000 tx=0x7fb71c030000 comp rx=0 tx=0).stop 2024-03-28T16:15:54.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.232+0000 7fb72d9ab640 10 monclient: shutdown 2024-03-28T16:15:54.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.232+0000 7fb72d9ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:54.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.232+0000 7fb72d9ab640 1 -- 172.21.15.7:0/446609966 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb728071480 msgr2=0x7fb72813faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.232+0000 7fb72d9ab640 1 --2- 172.21.15.7:0/446609966 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb728071480 0x7fb72813faf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb71800b470 tx=0x7fb71800b930 comp rx=0 tx=0).stop 2024-03-28T16:15:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.233+0000 7fb72d9ab640 1 -- 172.21.15.7:0/446609966 shutdown_connections 2024-03-28T16:15:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.233+0000 7fb72d9ab640 1 --2- 172.21.15.7:0/446609966 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7fb71406c5e0 0x7fb71406eaa0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.233+0000 7fb72d9ab640 1 --2- 172.21.15.7:0/446609966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7280b32e0 0x7fb728140710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.233+0000 7fb72d9ab640 1 --2- 172.21.15.7:0/446609966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb72813ee50 0x7fb7281d68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.233+0000 7fb72d9ab640 1 --2- 172.21.15.7:0/446609966 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb728071480 0x7fb72813faf0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.233+0000 7fb72d9ab640 1 -- 172.21.15.7:0/446609966 >> 172.21.15.7:0/446609966 conn(0x7fb7280b70e0 msgr2=0x7fb7280b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.233+0000 7fb72d9ab640 1 -- 172.21.15.7:0/446609966 shutdown_connections 2024-03-28T16:15:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.233+0000 7fb72d9ab640 1 -- 172.21.15.7:0/446609966 wait complete. 2024-03-28T16:15:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.233+0000 7fb72d9ab640 1 librados: shutdown 2024-03-28T16:15:54.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.238+0000 7f221f7fe640 1 -- 172.21.15.7:0/1499334056 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2230044090 con 0x7f22380b32e0 2024-03-28T16:15:54.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.238+0000 7f221f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-28T16:15:54.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.238+0000 7f221f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:54.239 INFO:teuthology.orchestra.run.smithi007.stdout:90194313227 2024-03-28T16:15:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.241+0000 7f223d7e4640 10 librados: watch_flush enter 2024-03-28T16:15:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.241+0000 7f223d7e4640 10 librados: watch_flush exit 2024-03-28T16:15:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.241+0000 7f223d7e4640 20 client.4359.objecter shutdown clearing up homeless session... 2024-03-28T16:15:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.241+0000 7f223d7e4640 10 client.4359.objecter successfully canceled tick 2024-03-28T16:15:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.241+0000 7f223d7e4640 1 -- 172.21.15.7:0/1499334056 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f222406c590 msgr2=0x7f222406ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.241+0000 7f223d7e4640 1 --2- 172.21.15.7:0/1499334056 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f222406c590 0x7f222406ea50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2228000f60 tx=0x7f22280079b0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.241+0000 7f223d7e4640 10 monclient: shutdown 2024-03-28T16:15:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.241+0000 7f223d7e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.241+0000 7f223d7e4640 1 -- 172.21.15.7:0/1499334056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22380b32e0 msgr2=0x7f2238143b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.241+0000 7f223d7e4640 1 --2- 172.21.15.7:0/1499334056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22380b32e0 0x7f2238143b40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2230037040 tx=0x7f2230004a40 comp rx=0 tx=0).stop 2024-03-28T16:15:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.241+0000 7f223d7e4640 1 -- 172.21.15.7:0/1499334056 shutdown_connections 2024-03-28T16:15:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.241+0000 7f223d7e4640 1 --2- 172.21.15.7:0/1499334056 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f222406c590 0x7f222406ea50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.241+0000 7f223d7e4640 1 --2- 172.21.15.7:0/1499334056 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f22380b32e0 0x7f2238143b40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.241+0000 7f223d7e4640 1 --2- 172.21.15.7:0/1499334056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22380a86e0 0x7f2238140770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.241+0000 7f223d7e4640 1 --2- 172.21.15.7:0/1499334056 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2238071480 0x7f223813d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.241+0000 7f223d7e4640 1 -- 172.21.15.7:0/1499334056 >> 172.21.15.7:0/1499334056 conn(0x7f22380b70e0 msgr2=0x7f22380b4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.242+0000 7f223d7e4640 1 -- 172.21.15.7:0/1499334056 shutdown_connections 2024-03-28T16:15:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.242+0000 7f223d7e4640 1 -- 172.21.15.7:0/1499334056 wait complete. 2024-03-28T16:15:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.242+0000 7f223d7e4640 1 librados: shutdown 2024-03-28T16:15:54.247 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.4 2024-03-28T16:15:54.247 DEBUG:teuthology.parallel:result is None 2024-03-28T16:15:54.254 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313227 got 90194313227 for osd.6 2024-03-28T16:15:54.254 DEBUG:teuthology.parallel:result is None 2024-03-28T16:15:54.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.408+0000 7efe144d5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-28T16:15:54.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.408+0000 7efe144d5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-28T16:15:54.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.408+0000 7efe144d5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:54.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.408+0000 7efe144d5640 1 -- 172.21.15.7:0/3709483386 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7efe0c069430 con 0x7efe0c071920 2024-03-28T16:15:54.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.412+0000 7efe02ffd640 1 -- 172.21.15.7:0/3709483386 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efe0807a1c0 con 0x7efe0c071920 2024-03-28T16:15:54.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.412+0000 7efe02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-28T16:15:54.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.412+0000 7efe02ffd640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:54.413 INFO:teuthology.orchestra.run.smithi007.stdout:73014444040 2024-03-28T16:15:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.414+0000 7efe144d5640 10 librados: watch_flush enter 2024-03-28T16:15:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.414+0000 7efe144d5640 10 librados: watch_flush exit 2024-03-28T16:15:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.414+0000 7efe144d5640 20 client.4307.objecter shutdown clearing up homeless session... 2024-03-28T16:15:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.414+0000 7efe144d5640 10 client.4307.objecter successfully canceled tick 2024-03-28T16:15:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.414+0000 7efe144d5640 1 -- 172.21.15.7:0/3709483386 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7efdf006c5e0 msgr2=0x7efdf006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.414+0000 7efe144d5640 1 --2- 172.21.15.7:0/3709483386 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7efdf006c5e0 0x7efdf006eaa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efdfc007fd0 tx=0x7efdfc007980 comp rx=0 tx=0).stop 2024-03-28T16:15:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.414+0000 7efe144d5640 10 monclient: shutdown 2024-03-28T16:15:54.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.414+0000 7efe144d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:54.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.414+0000 7efe144d5640 1 -- 172.21.15.7:0/3709483386 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7efe0c071920 msgr2=0x7efe0c141fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:54.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.414+0000 7efe144d5640 1 --2- 172.21.15.7:0/3709483386 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7efe0c071920 0x7efe0c141fa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efe08033000 tx=0x7efe080046c0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.414+0000 7efe144d5640 1 -- 172.21.15.7:0/3709483386 shutdown_connections 2024-03-28T16:15:54.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.414+0000 7efe144d5640 1 --2- 172.21.15.7:0/3709483386 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7efdf006c5e0 0x7efdf006eaa0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.414+0000 7efe144d5640 1 --2- 172.21.15.7:0/3709483386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efe0c13ed70 0x7efe0c13f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.414+0000 7efe144d5640 1 --2- 172.21.15.7:0/3709483386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe0c142850 0x7efe0c13f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.414+0000 7efe144d5640 1 --2- 172.21.15.7:0/3709483386 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7efe0c071920 0x7efe0c141fa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.414+0000 7efe144d5640 1 -- 172.21.15.7:0/3709483386 >> 172.21.15.7:0/3709483386 conn(0x7efe0c0b3db0 msgr2=0x7efe0c0b4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:54.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.414+0000 7efe144d5640 1 -- 172.21.15.7:0/3709483386 shutdown_connections 2024-03-28T16:15:54.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.414+0000 7efe144d5640 1 -- 172.21.15.7:0/3709483386 wait complete. 2024-03-28T16:15:54.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.414+0000 7efe144d5640 1 librados: shutdown 2024-03-28T16:15:54.426 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444040 got 73014444040 for osd.3 2024-03-28T16:15:54.426 DEBUG:teuthology.parallel:result is None 2024-03-28T16:15:54.426 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-28T16:15:54.426 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T16:15:54.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.525+0000 7f526ecf2640 10 monclient: get_monmap_and_config 2024-03-28T16:15:54.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.525+0000 7f526ecf2640 10 monclient: build_initial_monmap 2024-03-28T16:15:54.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.525+0000 7f526ecf2640 10 monclient: monmap: 2024-03-28T16:15:54.526 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:54.527152+0000 2024-03-28T16:15:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:54.527152+0000 2024-03-28T16:15:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:54.527 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:54.527 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:54.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.526+0000 7f526ecf2640 1 Processor -- start 2024-03-28T16:15:54.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.526+0000 7f526ecf2640 1 -- start start 2024-03-28T16:15:54.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.526+0000 7f526ecf2640 10 monclient: init 2024-03-28T16:15:54.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.526+0000 7f526ecf2640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:54.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.526+0000 7f526ecf2640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-03-28T16:15:54.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.526+0000 7f526ecf2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f526806a4d0 0x7f526806a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:54.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.526+0000 7f526ecf2640 10 monclient(hunting): picked mon.noname-c con 0x7f526806a4d0 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:54.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.526+0000 7f526ecf2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f526806b8c0 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:54.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.526+0000 7f526ecf2640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f52680afc20 0x7f52680b0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:54.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.526+0000 7f526ecf2640 10 monclient(hunting): picked mon.noname-a con 0x7f52680afc20 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:54.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.526+0000 7f526ecf2640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5268141d80 con 0x7f526806b8c0 2024-03-28T16:15:54.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.526+0000 7f526ecf2640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:54.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.526+0000 7f526ecf2640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52680b05b0 con 0x7f52680afc20 2024-03-28T16:15:54.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.526+0000 7f526ecf2640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:54.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.526+0000 7f526ecf2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f526806adf0 con 0x7f526806a4d0 2024-03-28T16:15:54.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.526+0000 7f526ecf2640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:54.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.526+0000 7f526ecf2640 10 monclient(hunting): authenticate will time out at 1421.668737s 2024-03-28T16:15:54.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.527+0000 7f526ca8e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f526806a4d0 0x7f526806a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:54.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.527+0000 7f526d28f640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f52680afc20 0x7f52680b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:54.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.527+0000 7f526ca8e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f526806a4d0 0x7f526806a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:58328/0 (socket says 172.21.15.7:58328) 2024-03-28T16:15:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.527+0000 7f526ca8e640 1 -- 172.21.15.7:0/1742322627 learned_addr learned my addr 172.21.15.7:0/1742322627 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.527+0000 7f526ca8e640 10 monclient(hunting): get_auth_request con 0x7f526806a4d0 auth_method 0 2024-03-28T16:15:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.527+0000 7f526ca8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.527+0000 7f526ca8e640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.527+0000 7f526ca8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.527+0000 7f526d28f640 10 monclient(hunting): get_auth_request con 0x7f52680afc20 auth_method 0 2024-03-28T16:15:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.527+0000 7f526d28f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.527+0000 7f526d28f640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.527+0000 7f526d28f640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.528+0000 7f526ca8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.528+0000 7f526ca8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:54.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.528+0000 7f526ca8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:54.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.529+0000 7f526d28f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.529+0000 7f526d28f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.529+0000 7f526d28f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.530+0000 7f526ca8e640 10 monclient(hunting): handle_auth_done global_id 4408 payload 274 2024-03-28T16:15:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.530+0000 7f526ca8e640 1 -- 172.21.15.7:0/1742322627 >> v2:172.21.15.7:3300/0 conn(0x7f52680afc20 msgr2=0x7f52680b0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.530+0000 7f526ca8e640 1 --2- 172.21.15.7:0/1742322627 >> v2:172.21.15.7:3300/0 conn(0x7f52680afc20 0x7f52680b0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.530+0000 7f526ca8e640 1 -- 172.21.15.7:0/1742322627 >> v1:172.21.15.7:6789/0 conn(0x7f526806b8c0 legacy=0x7f52680b0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.530+0000 7f526ca8e640 10 monclient: _finish_hunting 0 2024-03-28T16:15:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.530+0000 7f526ca8e640 1 monclient: found mon.noname-c 2024-03-28T16:15:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.530+0000 7f526ca8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.530+0000 7f526ca8e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3300/0 2024-03-28T16:15:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.530+0000 7f526ca8e640 1 -- 172.21.15.7:0/1742322627 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52681419e0 con 0x7f526806a4d0 2024-03-28T16:15:54.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.532+0000 7f526ca8e640 1 --2- 172.21.15.7:0/1742322627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f526806a4d0 0x7f526806a8b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f52680721d0 tx=0x7f525c02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e9c11b8b63138b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:54.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.535+0000 7f52677fe640 1 -- 172.21.15.7:0/1742322627 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525c031870 con 0x7f526806a4d0 2024-03-28T16:15:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.535+0000 7f52677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.535+0000 7f52677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.535+0000 7f52677fe640 10 monclient: dump: 2024-03-28T16:15:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:54.538 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:54.539 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:54.539 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:54.539 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:54.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.535+0000 7f52677fe640 10 monclient: _finish_auth 0 2024-03-28T16:15:54.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.535+0000 7f52677fe640 10 monclient: _check_auth_tickets 2024-03-28T16:15:54.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.535+0000 7f52677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:54.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.535+0000 7f52677fe640 1 -- 172.21.15.7:0/1742322627 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f525c0319d0 con 0x7f526806a4d0 2024-03-28T16:15:54.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.535+0000 7f52677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:54.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.535+0000 7f52677fe640 1 -- 172.21.15.7:0/1742322627 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525c031d60 con 0x7f526806a4d0 2024-03-28T16:15:54.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.535+0000 7f526ecf2640 5 monclient: authenticate success, global_id 4408 2024-03-28T16:15:54.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.535+0000 7f526ecf2640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:54.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.535+0000 7f526ecf2640 4 set_mon_vals no callback set 2024-03-28T16:15:54.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.535+0000 7f52677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:54.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.535+0000 7f52677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:54.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.535+0000 7f52677fe640 10 monclient: dump: 2024-03-28T16:15:54.540 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:54.540 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:54.540 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:54.540 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:54.540 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:54.540 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:54.540 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:54.540 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:54.540 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:54.540 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:54.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.536+0000 7f526ecf2640 10 monclient: shutdown 2024-03-28T16:15:54.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.536+0000 7f526ecf2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:54.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.536+0000 7f526ecf2640 1 -- 172.21.15.7:0/1742322627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f526806a4d0 msgr2=0x7f526806a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:54.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.536+0000 7f526ecf2640 1 --2- 172.21.15.7:0/1742322627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f526806a4d0 0x7f526806a8b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f52680721d0 tx=0x7f525c02b910 comp rx=0 tx=0).stop 2024-03-28T16:15:54.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.536+0000 7f526ecf2640 1 -- 172.21.15.7:0/1742322627 shutdown_connections 2024-03-28T16:15:54.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.536+0000 7f526ecf2640 1 --2- 172.21.15.7:0/1742322627 >> v2:172.21.15.7:3300/0 conn(0x7f52680afc20 0x7f52680b0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.536+0000 7f526ecf2640 1 --2- 172.21.15.7:0/1742322627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f526806a4d0 0x7f526806a8b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.536+0000 7f526ecf2640 1 -- 172.21.15.7:0/1742322627 >> 172.21.15.7:0/1742322627 conn(0x7f52680b5220 msgr2=0x7f52680b5630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:54.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.536+0000 7f526ecf2640 1 -- 172.21.15.7:0/1742322627 shutdown_connections 2024-03-28T16:15:54.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.536+0000 7f526ecf2640 1 -- 172.21.15.7:0/1742322627 wait complete. 2024-03-28T16:15:54.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.537+0000 7f526ecf2640 10 monclient: build_initial_monmap 2024-03-28T16:15:54.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.537+0000 7f526ecf2640 10 monclient: monmap: 2024-03-28T16:15:54.545 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:54.545 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:54.545 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:54.545 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:54.545 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:54.545 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:54.545 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:54.545 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:54.545 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:54.545 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:54.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.537+0000 7f526ecf2640 1 librados: starting msgr at 2024-03-28T16:15:54.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.537+0000 7f526ecf2640 1 librados: starting objecter 2024-03-28T16:15:54.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.537+0000 7f526ecf2640 1 Processor -- start 2024-03-28T16:15:54.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.537+0000 7f526ecf2640 1 -- start start 2024-03-28T16:15:54.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.537+0000 7f526ecf2640 1 librados: setting wanted keys 2024-03-28T16:15:54.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.537+0000 7f526ecf2640 1 librados: calling monclient init 2024-03-28T16:15:54.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.537+0000 7f526ecf2640 10 monclient: init 2024-03-28T16:15:54.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.538+0000 7f526ecf2640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:54.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.538+0000 7f526ecf2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-28T16:15:54.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.538+0000 7f526ecf2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f526806a4d0 0x7f526814b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:54.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.538+0000 7f526ecf2640 10 monclient(hunting): picked mon.noname-b con 0x7f526806a4d0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:54.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.538+0000 7f526ecf2640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f526806b8c0 0x7f5268143c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:54.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.538+0000 7f526ecf2640 10 monclient(hunting): picked mon.noname-a con 0x7f526806b8c0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:54.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.538+0000 7f526ecf2640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f52680afc20 0x7f52681d9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:54.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.538+0000 7f526ecf2640 10 monclient(hunting): picked mon.noname-c con 0x7f52680afc20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:54.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.538+0000 7f526ecf2640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:54.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.538+0000 7f526ecf2640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52681442c0 con 0x7f526806b8c0 2024-03-28T16:15:54.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.538+0000 7f526ecf2640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:54.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.538+0000 7f526ecf2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5268144510 con 0x7f526806a4d0 2024-03-28T16:15:54.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.538+0000 7f526ecf2640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:54.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.538+0000 7f526ecf2640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5268142c00 con 0x7f52680afc20 2024-03-28T16:15:54.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.538+0000 7f526ecf2640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:54.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.538+0000 7f526ecf2640 10 monclient(hunting): authenticate will time out at 1421.680259s 2024-03-28T16:15:54.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.538+0000 7f5267fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f526806b8c0 0x7f5268143c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:54.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.538+0000 7f526d28f640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f52680afc20 0x7f52681d9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:54.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.538+0000 7f526ca8e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f526806a4d0 0x7f526814b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:54.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.539+0000 7f526ca8e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f526806a4d0 0x7f526814b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:58344/0 (socket says 172.21.15.7:58344) 2024-03-28T16:15:54.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.539+0000 7f526ca8e640 1 -- 172.21.15.7:0/1305566695 learned_addr learned my addr 172.21.15.7:0/1305566695 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:54.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.539+0000 7f526ca8e640 10 monclient(hunting): get_auth_request con 0x7f526806a4d0 auth_method 0 2024-03-28T16:15:54.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.539+0000 7f526ca8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:54.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.539+0000 7f526ca8e640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:54.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.539+0000 7f526ca8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:54.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.539+0000 7f526d28f640 10 monclient(hunting): get_auth_request con 0x7f52680afc20 auth_method 0 2024-03-28T16:15:54.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.539+0000 7f526d28f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:54.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.539+0000 7f526d28f640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:54.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.539+0000 7f526d28f640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:54.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.539+0000 7f5267fff640 10 monclient(hunting): get_auth_request con 0x7f526806b8c0 auth_method 0 2024-03-28T16:15:54.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.539+0000 7f5267fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:54.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.539+0000 7f5267fff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:54.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.539+0000 7f5267fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:54.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.539+0000 7f526ca8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:54.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.539+0000 7f526ca8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:54.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.539+0000 7f526ca8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:54.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.540+0000 7f526d28f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:54.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.540+0000 7f526d28f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:54.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.540+0000 7f526d28f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:54.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.540+0000 7f5267fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:54.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.540+0000 7f5267fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:54.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.540+0000 7f5267fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:54.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.542+0000 7f526ca8e640 10 monclient(hunting): handle_auth_done global_id 4411 payload 824 2024-03-28T16:15:54.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.542+0000 7f526ca8e640 1 -- 172.21.15.7:0/1305566695 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f52680afc20 msgr2=0x7f52681d9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:54.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.542+0000 7f526ca8e640 1 --2- 172.21.15.7:0/1305566695 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f52680afc20 0x7f52681d9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.542+0000 7f526ca8e640 1 -- 172.21.15.7:0/1305566695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f526806b8c0 msgr2=0x7f5268143c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:54.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.542+0000 7f526ca8e640 1 --2- 172.21.15.7:0/1305566695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f526806b8c0 0x7f5268143c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:54.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.542+0000 7f526ca8e640 10 monclient: _finish_hunting 0 2024-03-28T16:15:54.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.542+0000 7f526ca8e640 1 monclient: found mon.noname-b 2024-03-28T16:15:54.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.542+0000 7f526ca8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:54.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.542+0000 7f526ca8e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:15:54.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.542+0000 7f526ca8e640 1 -- 172.21.15.7:0/1305566695 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525c0094d0 con 0x7f526806a4d0 2024-03-28T16:15:54.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.544+0000 7f526ca8e640 1 --2- 172.21.15.7:0/1305566695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f526806a4d0 0x7f526814b620 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f525c02b950 tx=0x7f525c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:54.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.544+0000 7f52657fa640 10 client.?.objecter ms_handle_connect 0x7f526806a4d0 2024-03-28T16:15:54.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.544+0000 7f52657fa640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:54.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f52657fa640 1 -- 172.21.15.7:0/1305566695 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525c03abe0 con 0x7f526806a4d0 2024-03-28T16:15:54.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f52657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:54.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f52657fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:54.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f52657fa640 10 monclient: dump: 2024-03-28T16:15:54.551 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:54.551 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:54.551 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:54.552 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:54.552 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:54.552 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:54.552 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:54.552 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:54.552 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:54.552 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:54.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f52657fa640 10 monclient: _finish_auth 0 2024-03-28T16:15:54.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f52657fa640 10 monclient: _check_auth_tickets 2024-03-28T16:15:54.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f52657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:54.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f52657fa640 1 -- 172.21.15.7:0/1305566695 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f525c03ad60 con 0x7f526806a4d0 2024-03-28T16:15:54.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f52657fa640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:54.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f52657fa640 1 -- 172.21.15.7:0/1305566695 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525c044630 con 0x7f526806a4d0 2024-03-28T16:15:54.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f526ecf2640 5 monclient: authenticate success, global_id 4411 2024-03-28T16:15:54.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f52657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:54.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f52657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:54.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f52657fa640 10 monclient: dump: 2024-03-28T16:15:54.553 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:54.553 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:54.553 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:54.553 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:54.553 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:54.553 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:54.553 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:54.553 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:54.553 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:54.553 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:54.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f526ecf2640 10 monclient: _renew_subs 2024-03-28T16:15:54.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f526ecf2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:54.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f5266ffd640 4 set_mon_vals no callback set 2024-03-28T16:15:54.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f526ecf2640 1 -- 172.21.15.7:0/1305566695 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5268145830 con 0x7f526806a4d0 2024-03-28T16:15:54.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f526ecf2640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:54.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f526ecf2640 10 monclient: _renew_subs 2024-03-28T16:15:54.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f526ecf2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:54.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f526ecf2640 1 -- 172.21.15.7:0/1305566695 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5268144c10 con 0x7f526806a4d0 2024-03-28T16:15:54.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.548+0000 7f526ecf2640 1 librados: init done 2024-03-28T16:15:54.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.549+0000 7f526ecf2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:54.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.549+0000 7f526ecf2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:54.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.549+0000 7f526ecf2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:54.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.549+0000 7f526ecf2640 1 -- 172.21.15.7:0/1305566695 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5268069430 con 0x7f526806a4d0 2024-03-28T16:15:54.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.790+0000 7f52657fa640 1 -- 172.21.15.7:0/1305566695 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f525c04d450 con 0x7f526806a4d0 2024-03-28T16:15:54.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.790+0000 7f52657fa640 1 --2- 172.21.15.7:0/1305566695 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f524c06c590 0x7f524c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:54.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.790+0000 7f5267fff640 1 --2- 172.21.15.7:0/1305566695 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f524c06c590 0x7f524c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:54.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.790+0000 7f5267fff640 10 monclient: get_auth_request con 0x7f524c06c590 auth_method 0 2024-03-28T16:15:54.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.791+0000 7f5267fff640 1 --2- 172.21.15.7:0/1305566695 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f524c06c590 0x7f524c06ea50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f526813ace0 tx=0x7f5254009040 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:54.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.791+0000 7f52657fa640 10 client.4411.objecter ms_handle_connect 0x7f524c06c590 2024-03-28T16:15:54.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.792+0000 7f52657fa640 1 -- 172.21.15.7:0/1305566695 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f525c02d040 con 0x7f526806a4d0 2024-03-28T16:15:54.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.792+0000 7f52657fa640 10 client.4411.objecter ms_dispatch 0x7f5268000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:54.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.792+0000 7f52657fa640 3 client.4411.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:54.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.792+0000 7f52657fa640 3 client.4411.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:54.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.792+0000 7f52657fa640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-28T16:15:54.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.810+0000 7f52657fa640 1 -- 172.21.15.7:0/1305566695 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f525c076f20 con 0x7f526806a4d0 2024-03-28T16:15:54.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.810+0000 7f52657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:54.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.810+0000 7f52657fa640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:54.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.993+0000 7f526ecf2640 1 -- 172.21.15.7:0/1305566695 --> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f52680b41f0 con 0x7f524c06c590 2024-03-28T16:15:54.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.996+0000 7f52657fa640 1 -- 172.21.15.7:0/1305566695 <== mgr.4162 v2:172.21.15.83:6800/34752 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48010 (secure 0 0 0) 0x7f52680b41f0 con 0x7f524c06c590 2024-03-28T16:15:54.997 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:15:54.997 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-28T16:15:54.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.997+0000 7f526ecf2640 10 librados: watch_flush enter 2024-03-28T16:15:54.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.997+0000 7f526ecf2640 10 librados: watch_flush exit 2024-03-28T16:15:54.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.997+0000 7f526ecf2640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-28T16:15:54.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.997+0000 7f526ecf2640 10 client.4411.objecter successfully canceled tick 2024-03-28T16:15:54.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.997+0000 7f526ecf2640 1 -- 172.21.15.7:0/1305566695 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f524c06c590 msgr2=0x7f524c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:54.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.997+0000 7f526ecf2640 1 --2- 172.21.15.7:0/1305566695 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f524c06c590 0x7f524c06ea50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f526813ace0 tx=0x7f5254009040 comp rx=0 tx=0).stop 2024-03-28T16:15:54.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.997+0000 7f526ecf2640 10 monclient: shutdown 2024-03-28T16:15:55.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.997+0000 7f526ecf2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:55.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.997+0000 7f526ecf2640 1 -- 172.21.15.7:0/1305566695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f526806a4d0 msgr2=0x7f526814b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:55.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.997+0000 7f526ecf2640 1 --2- 172.21.15.7:0/1305566695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f526806a4d0 0x7f526814b620 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f525c02b950 tx=0x7f525c0046c0 comp rx=0 tx=0).stop 2024-03-28T16:15:55.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.997+0000 7f526ecf2640 1 -- 172.21.15.7:0/1305566695 shutdown_connections 2024-03-28T16:15:55.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.997+0000 7f526ecf2640 1 --2- 172.21.15.7:0/1305566695 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f524c06c590 0x7f524c06ea50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:55.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.997+0000 7f526ecf2640 1 --2- 172.21.15.7:0/1305566695 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f526806b8c0 0x7f5268143c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:55.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.997+0000 7f526ecf2640 1 --2- 172.21.15.7:0/1305566695 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f52680afc20 0x7f52681d9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:55.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.997+0000 7f526ecf2640 1 --2- 172.21.15.7:0/1305566695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f526806a4d0 0x7f526814b620 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:55.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.997+0000 7f526ecf2640 1 -- 172.21.15.7:0/1305566695 >> 172.21.15.7:0/1305566695 conn(0x7f52680b5220 msgr2=0x7f5268068ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:55.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.997+0000 7f526ecf2640 1 -- 172.21.15.7:0/1305566695 shutdown_connections 2024-03-28T16:15:55.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.997+0000 7f526ecf2640 1 -- 172.21.15.7:0/1305566695 wait complete. 2024-03-28T16:15:55.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:54.997+0000 7f526ecf2640 1 librados: shutdown 2024-03-28T16:15:55.009 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-03-28T16:15:53.041724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219180,"kb_used_data":3012,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755540,"statfs":{"total":773094113280,"available":772869672960,"internally_reserved":0,"allocated":3084288,"data_stored":1573996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":117,"apply_latency_ms":117,"commit_latency_ns":117000000,"apply_latency_ns":117000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.451511"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-03-28T16:15:49.154143+0000","last_change":"2024-03-28T16:15:49.142733+0000","last_active":"2024-03-28T16:15:49.154143+0000","last_peered":"2024-03-28T16:15:49.154143+0000","last_clean":"2024-03-28T16:15:49.154143+0000","last_became_active":"2024-03-28T16:15:49.135276+0000","last_became_peered":"2024-03-28T16:15:49.135276+0000","last_unstale":"2024-03-28T16:15:49.154143+0000","last_undegraded":"2024-03-28T16:15:49.154143+0000","last_fullsized":"2024-03-28T16:15:49.154143+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_clean_scrub_stamp":"2024-03-28T16:15:46.495373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:12:40.798563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-03-28T16:15:49.196738+0000","last_change":"2024-03-28T16:15:49.179203+0000","last_active":"2024-03-28T16:15:49.196738+0000","last_peered":"2024-03-28T16:15:49.196738+0000","last_clean":"2024-03-28T16:15:49.196738+0000","last_became_active":"2024-03-28T16:15:49.153269+0000","last_became_peered":"2024-03-28T16:15:49.153269+0000","last_unstale":"2024-03-28T16:15:49.196738+0000","last_undegraded":"2024-03-28T16:15:49.196738+0000","last_fullsized":"2024-03-28T16:15:49.196738+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_clean_scrub_stamp":"2024-03-28T16:15:46.495373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:16.114251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-03-28T16:15:48.613882+0000","last_change":"2024-03-28T16:15:48.601259+0000","last_active":"2024-03-28T16:15:48.613882+0000","last_peered":"2024-03-28T16:15:48.613882+0000","last_clean":"2024-03-28T16:15:48.613882+0000","last_became_active":"2024-03-28T16:15:48.596033+0000","last_became_peered":"2024-03-28T16:15:48.596033+0000","last_unstale":"2024-03-28T16:15:48.613882+0000","last_undegraded":"2024-03-28T16:15:48.613882+0000","last_fullsized":"2024-03-28T16:15:48.613882+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_clean_scrub_stamp":"2024-03-28T16:15:46.495373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:39:38.041200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-03-28T16:15:50.047529+0000","last_change":"2024-03-28T16:15:48.819449+0000","last_active":"2024-03-28T16:15:50.047529+0000","last_peered":"2024-03-28T16:15:50.047529+0000","last_clean":"2024-03-28T16:15:50.047529+0000","last_became_active":"2024-03-28T16:15:48.790674+0000","last_became_peered":"2024-03-28T16:15:48.790674+0000","last_unstale":"2024-03-28T16:15:50.047529+0000","last_undegraded":"2024-03-28T16:15:50.047529+0000","last_fullsized":"2024-03-28T16:15:50.047529+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_clean_scrub_stamp":"2024-03-28T16:15:46.495373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:33:30.273205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-03-28T16:15:48.783171+0000","last_change":"2024-03-28T16:15:48.773840+0000","last_active":"2024-03-28T16:15:48.783171+0000","last_peered":"2024-03-28T16:15:48.783171+0000","last_clean":"2024-03-28T16:15:48.783171+0000","last_became_active":"2024-03-28T16:15:48.764041+0000","last_became_peered":"2024-03-28T16:15:48.764041+0000","last_unstale":"2024-03-28T16:15:48.783171+0000","last_undegraded":"2024-03-28T16:15:48.783171+0000","last_fullsized":"2024-03-28T16:15:48.783171+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_clean_scrub_stamp":"2024-03-28T16:15:46.495373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:51:37.011076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-03-28T16:15:50.049241+0000","last_change":"2024-03-28T16:15:48.502499+0000","last_active":"2024-03-28T16:15:50.049241+0000","last_peered":"2024-03-28T16:15:50.049241+0000","last_clean":"2024-03-28T16:15:50.049241+0000","last_became_active":"2024-03-28T16:15:48.474531+0000","last_became_peered":"2024-03-28T16:15:48.474531+0000","last_unstale":"2024-03-28T16:15:50.049241+0000","last_undegraded":"2024-03-28T16:15:50.049241+0000","last_fullsized":"2024-03-28T16:15:50.049241+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_clean_scrub_stamp":"2024-03-28T16:15:46.495373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:11:08.194988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-03-28T16:15:48.611809+0000","last_change":"2024-03-28T16:15:48.549497+0000","last_active":"2024-03-28T16:15:48.611809+0000","last_peered":"2024-03-28T16:15:48.611809+0000","last_clean":"2024-03-28T16:15:48.611809+0000","last_became_active":"2024-03-28T16:15:48.521388+0000","last_became_peered":"2024-03-28T16:15:48.521388+0000","last_unstale":"2024-03-28T16:15:48.611809+0000","last_undegraded":"2024-03-28T16:15:48.611809+0000","last_fullsized":"2024-03-28T16:15:48.611809+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_clean_scrub_stamp":"2024-03-28T16:15:46.495373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:06.462972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-03-28T16:15:48.853506+0000","last_change":"2024-03-28T16:15:48.847000+0000","last_active":"2024-03-28T16:15:48.853506+0000","last_peered":"2024-03-28T16:15:48.853506+0000","last_clean":"2024-03-28T16:15:48.853506+0000","last_became_active":"2024-03-28T16:15:48.841691+0000","last_became_peered":"2024-03-28T16:15:48.841691+0000","last_unstale":"2024-03-28T16:15:48.853506+0000","last_undegraded":"2024-03-28T16:15:48.853506+0000","last_fullsized":"2024-03-28T16:15:48.853506+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_clean_scrub_stamp":"2024-03-28T16:15:46.495373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:09:19.425223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":133,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-03-28T16:15:48.188781+0000","last_change":"2024-03-28T16:15:45.207007+0000","last_active":"2024-03-28T16:15:48.188781+0000","last_peered":"2024-03-28T16:15:48.188781+0000","last_clean":"2024-03-28T16:15:48.188781+0000","last_became_active":"2024-03-28T16:15:44.645475+0000","last_became_peered":"2024-03-28T16:15:44.645475+0000","last_unstale":"2024-03-28T16:15:48.188781+0000","last_undegraded":"2024-03-28T16:15:48.188781+0000","last_fullsized":"2024-03-28T16:15:48.188781+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:15:31.785347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:15:31.785347+0000","last_clean_scrub_stamp":"2024-03-28T16:15:31.785347+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:19:25.283430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":23,"seq":98784247816,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":688128,"data_stored":495848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4489999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4949999999999992}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.786}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.372}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.223000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.813000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.807}]}]},{"osd":6,"up_from":21,"seq":90194313227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":92260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.713999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.949000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":113.253}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":51.186}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":56.518999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":59.186999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":51.963999999999999}]}]},{"osd":3,"up_from":17,"seq":73014444040,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":98008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.5}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.3959999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.417}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.43}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.379999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.968}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.553999999999998}]}]},{"osd":2,"up_from":17,"seq":73014444040,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":98008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.877000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0279999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.282999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.246}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.613}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.73}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.621}]}]},{"osd":1,"up_from":17,"seq":73014444038,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":688128,"data_stored":495848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.605}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8140000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":61.380000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.196}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.217000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.089}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.711}]}]},{"osd":7,"up_from":15,"seq":64424509448,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":98008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.137}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0860000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.57}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":202.93299999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.0540000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.91}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.513999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":98008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.696999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0719999999999992}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.706}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.725999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.757999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":30.998999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":32.948}]}]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":98008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":30.809999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.561999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.042}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.746}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4139999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.112}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.837}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T16:15:55.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T16:15:55.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.110+0000 7f97bcda4640 10 monclient: get_monmap_and_config 2024-03-28T16:15:55.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.110+0000 7f97bcda4640 10 monclient: build_initial_monmap 2024-03-28T16:15:55.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.110+0000 7f97bcda4640 10 monclient: monmap: 2024-03-28T16:15:55.112 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:55.112 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:55.112 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:55.112105+0000 2024-03-28T16:15:55.112 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:55.112105+0000 2024-03-28T16:15:55.112 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:55.112 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:55.112 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:55.112 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:55.113 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:55.113 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:55.113 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:55.113 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:55.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.111+0000 7f97bcda4640 1 Processor -- start 2024-03-28T16:15:55.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.111+0000 7f97bcda4640 1 -- start start 2024-03-28T16:15:55.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.111+0000 7f97bcda4640 10 monclient: init 2024-03-28T16:15:55.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.111+0000 7f97bcda4640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:55.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.111+0000 7f97bcda4640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-03-28T16:15:55.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.111+0000 7f97bcda4640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f97b8071920 0x7f97b813e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:55.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.111+0000 7f97bcda4640 10 monclient(hunting): picked mon.noname-b con 0x7f97b8071920 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:55.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.111+0000 7f97bcda4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97b80a89a0 addr v1:172.21.15.7:6789/0 2024-03-28T16:15:55.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.111+0000 7f97bcda4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97b813ed70 addr v1:172.21.15.83:6789/0 2024-03-28T16:15:55.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.111+0000 7f97bcda4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b80a8ec0 con 0x7f97b80a89a0 2024-03-28T16:15:55.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.111+0000 7f97bcda4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b80aa660 con 0x7f97b813ed70 2024-03-28T16:15:55.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.111+0000 7f97bcda4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:55.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.111+0000 7f97bcda4640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b80aa480 con 0x7f97b8071920 2024-03-28T16:15:55.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.111+0000 7f97bcda4640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:55.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.111+0000 7f97bcda4640 10 monclient(hunting): authenticate will time out at 1422.253705s 2024-03-28T16:15:55.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.112+0000 7f97b659c640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f97b8071920 0x7f97b813e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:55.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.112+0000 7f97b6d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f97b813ed70 0x7f97b814bde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.7:42456/0 (socket says 172.21.15.7:42456) 2024-03-28T16:15:55.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.112+0000 7f97b6d9d640 1 -- 172.21.15.7:0/3254249317 learned_addr learned my addr 172.21.15.7:0/3254249317 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-28T16:15:55.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.112+0000 7f97b659c640 10 monclient(hunting): get_auth_request con 0x7f97b8071920 auth_method 0 2024-03-28T16:15:55.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.112+0000 7f97b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:55.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.112+0000 7f97b659c640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:55.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.112+0000 7f97b659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:55.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.113+0000 7f97b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:55.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.113+0000 7f97b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:55.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.113+0000 7f97b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:55.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.114+0000 7f97b559a640 1 -- 172.21.15.7:0/3254249317 <== mon.1 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725797512 0 0) 0x7f97b80aa660 con 0x7f97b813ed70 2024-03-28T16:15:55.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.114+0000 7f97b559a640 10 monclient(hunting): _negotiate 2024-03-28T16:15:55.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.114+0000 7f97b559a640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:55.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.114+0000 7f97b559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:55.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.114+0000 7f97b559a640 10 monclient(hunting): my global_id is 4414 2024-03-28T16:15:55.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.114+0000 7f97b559a640 1 -- 172.21.15.7:0/3254249317 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9798003520 con 0x7f97b813ed70 2024-03-28T16:15:55.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.115+0000 7f97b559a640 1 -- 172.21.15.7:0/3254249317 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658341262 0 0) 0x7f97b80a8ec0 con 0x7f97b80a89a0 2024-03-28T16:15:55.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.115+0000 7f97b559a640 10 monclient(hunting): _negotiate 2024-03-28T16:15:55.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.115+0000 7f97b559a640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:55.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.115+0000 7f97b559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:55.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.115+0000 7f97b559a640 10 monclient(hunting): my global_id is 4371 2024-03-28T16:15:55.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.115+0000 7f97b559a640 1 -- 172.21.15.7:0/3254249317 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97b80aa660 con 0x7f97b80a89a0 2024-03-28T16:15:55.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.117+0000 7f97b659c640 10 monclient(hunting): handle_auth_done global_id 4313 payload 274 2024-03-28T16:15:55.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.117+0000 7f97b659c640 1 -- 172.21.15.7:0/3254249317 >> v1:172.21.15.83:6789/0 conn(0x7f97b813ed70 legacy=0x7f97b814bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:55.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.117+0000 7f97b659c640 1 -- 172.21.15.7:0/3254249317 >> v1:172.21.15.7:6789/0 conn(0x7f97b80a89a0 legacy=0x7f97b80b1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:55.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.118+0000 7f97b659c640 10 monclient: _finish_hunting 0 2024-03-28T16:15:55.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.118+0000 7f97b659c640 1 monclient: found mon.noname-b 2024-03-28T16:15:55.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.118+0000 7f97b659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:55.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.118+0000 7f97b659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2024-03-28T16:15:55.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.118+0000 7f97b659c640 1 -- 172.21.15.7:0/3254249317 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b80b36d0 con 0x7f97b8071920 2024-03-28T16:15:55.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.119+0000 7f97b659c640 1 --2- 172.21.15.7:0/3254249317 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f97b8071920 0x7f97b813e830 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f97a00097d0 tx=0x7f97a002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2167b51639a53fec server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:55.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.123+0000 7f97b559a640 1 -- 172.21.15.7:0/3254249317 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a0031800 con 0x7f97b8071920 2024-03-28T16:15:55.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.123+0000 7f97b559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:55.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.123+0000 7f97b559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:55.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.123+0000 7f97b559a640 10 monclient: dump: 2024-03-28T16:15:55.125 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:55.125 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:55.125 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:55.125 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:55.125 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:55.125 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:55.125 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:55.125 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:55.126 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:55.126 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:55.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97b559a640 10 monclient: _finish_auth 0 2024-03-28T16:15:55.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97b559a640 10 monclient: _check_auth_tickets 2024-03-28T16:15:55.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97b559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:55.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97b559a640 1 -- 172.21.15.7:0/3254249317 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a0031960 con 0x7f97b8071920 2024-03-28T16:15:55.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97bcda4640 5 monclient: authenticate success, global_id 4313 2024-03-28T16:15:55.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97bcda4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:55.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97b559a640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:55.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97bcda4640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:55.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97b559a640 1 -- 172.21.15.7:0/3254249317 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a0031cf0 con 0x7f97b8071920 2024-03-28T16:15:55.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97b559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:55.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97bcda4640 4 set_mon_vals no callback set 2024-03-28T16:15:55.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97b559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:15:55.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97b559a640 10 monclient: dump: 2024-03-28T16:15:55.127 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:55.127 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:55.127 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:55.127 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:55.127 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:55.127 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:55.127 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:55.127 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:55.127 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:55.127 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:55.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97bcda4640 10 monclient: shutdown 2024-03-28T16:15:55.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97bcda4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:55.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97bcda4640 1 -- 172.21.15.7:0/3254249317 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f97b8071920 msgr2=0x7f97b813e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:55.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97bcda4640 1 --2- 172.21.15.7:0/3254249317 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f97b8071920 0x7f97b813e830 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f97a00097d0 tx=0x7f97a002b8a0 comp rx=0 tx=0).stop 2024-03-28T16:15:55.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97bcda4640 1 -- 172.21.15.7:0/3254249317 shutdown_connections 2024-03-28T16:15:55.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97bcda4640 1 --2- 172.21.15.7:0/3254249317 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f97b8071920 0x7f97b813e830 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:55.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97bcda4640 1 -- 172.21.15.7:0/3254249317 >> 172.21.15.7:0/3254249317 conn(0x7f97b80b3db0 msgr2=0x7f97b80b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:55.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97bcda4640 1 -- 172.21.15.7:0/3254249317 shutdown_connections 2024-03-28T16:15:55.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.124+0000 7f97bcda4640 1 -- 172.21.15.7:0/3254249317 wait complete. 2024-03-28T16:15:55.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.125+0000 7f97bcda4640 10 monclient: build_initial_monmap 2024-03-28T16:15:55.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.125+0000 7f97bcda4640 10 monclient: monmap: 2024-03-28T16:15:55.132 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:55.132 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:55.132 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:55.132 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:55.132 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:55.132 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:55.132 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:55.132 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:55.132 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:55.132 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:55.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 1 librados: starting msgr at 2024-03-28T16:15:55.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 1 librados: starting objecter 2024-03-28T16:15:55.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 1 Processor -- start 2024-03-28T16:15:55.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 1 -- start start 2024-03-28T16:15:55.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 1 librados: setting wanted keys 2024-03-28T16:15:55.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 1 librados: calling monclient init 2024-03-28T16:15:55.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 10 monclient: init 2024-03-28T16:15:55.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:55.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-28T16:15:55.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97b8071920 0x7f97b8145450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:55.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 10 monclient(hunting): picked mon.noname-a con 0x7f97b8071920 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:55.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f97b813ed70 0x7f97b813f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:55.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 10 monclient(hunting): picked mon.noname-c con 0x7f97b813ed70 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:55.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97b8140d40 0x7f97b81d72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:55.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 10 monclient(hunting): picked mon.noname-b con 0x7f97b8140d40 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:55.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:55.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b81d6aa0 con 0x7f97b8071920 2024-03-28T16:15:55.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:55.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b81d6d80 con 0x7f97b8140d40 2024-03-28T16:15:55.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:55.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b81411d0 con 0x7f97b813ed70 2024-03-28T16:15:55.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:55.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.126+0000 7f97bcda4640 10 monclient(hunting): authenticate will time out at 1422.268618s 2024-03-28T16:15:55.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.127+0000 7f97b5d9b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f97b813ed70 0x7f97b813f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:55.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.127+0000 7f97b659c640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97b8071920 0x7f97b8145450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:55.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.127+0000 7f97b6d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97b8140d40 0x7f97b81d72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:55.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.127+0000 7f97b5d9b640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f97b813ed70 0x7f97b813f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:60428/0 (socket says 172.21.15.7:60428) 2024-03-28T16:15:55.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.127+0000 7f97b5d9b640 1 -- 172.21.15.7:0/2879071176 learned_addr learned my addr 172.21.15.7:0/2879071176 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:55.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.127+0000 7f97b5d9b640 10 monclient(hunting): get_auth_request con 0x7f97b813ed70 auth_method 0 2024-03-28T16:15:55.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.127+0000 7f97b5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:55.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.127+0000 7f97b5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:55.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.127+0000 7f97b5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:55.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.127+0000 7f97b6d9d640 10 monclient(hunting): get_auth_request con 0x7f97b8140d40 auth_method 0 2024-03-28T16:15:55.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.127+0000 7f97b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:55.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.127+0000 7f97b6d9d640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:55.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.127+0000 7f97b6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:55.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.127+0000 7f97b659c640 10 monclient(hunting): get_auth_request con 0x7f97b8071920 auth_method 0 2024-03-28T16:15:55.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.127+0000 7f97b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:55.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.127+0000 7f97b659c640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:55.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.127+0000 7f97b659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:55.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.128+0000 7f97b5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:55.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.128+0000 7f97b5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:55.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.128+0000 7f97b5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:55.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.128+0000 7f97b6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:55.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.128+0000 7f97b6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:55.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.128+0000 7f97b6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:55.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.128+0000 7f97b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:55.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.128+0000 7f97b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:55.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.128+0000 7f97b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:55.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.130+0000 7f97b5d9b640 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2024-03-28T16:15:55.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.130+0000 7f97b5d9b640 1 -- 172.21.15.7:0/2879071176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97b8140d40 msgr2=0x7f97b81d72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:55.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.130+0000 7f97b5d9b640 1 --2- 172.21.15.7:0/2879071176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97b8140d40 0x7f97b81d72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:55.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.130+0000 7f97b5d9b640 1 -- 172.21.15.7:0/2879071176 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97b8071920 msgr2=0x7f97b8145450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:55.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.130+0000 7f97b5d9b640 1 --2- 172.21.15.7:0/2879071176 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97b8071920 0x7f97b8145450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:55.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.130+0000 7f97b5d9b640 10 monclient: _finish_hunting 0 2024-03-28T16:15:55.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.130+0000 7f97b5d9b640 1 monclient: found mon.noname-c 2024-03-28T16:15:55.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.130+0000 7f97b5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:55.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.130+0000 7f97b5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2024-03-28T16:15:55.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.130+0000 7f97b5d9b640 1 -- 172.21.15.7:0/2879071176 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a0009400 con 0x7f97b813ed70 2024-03-28T16:15:55.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.133+0000 7f97b5d9b640 1 --2- 172.21.15.7:0/2879071176 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f97b813ed70 0x7f97b813f350 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f97b000c800 tx=0x7f97b000ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:55.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.133+0000 7f97aeffd640 10 client.?.objecter ms_handle_connect 0x7f97b813ed70 2024-03-28T16:15:55.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.133+0000 7f97aeffd640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:55.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.136+0000 7f97aeffd640 1 -- 172.21.15.7:0/2879071176 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b0003b70 con 0x7f97b813ed70 2024-03-28T16:15:55.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.136+0000 7f97aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:55.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.136+0000 7f97aeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-28T16:15:55.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.136+0000 7f97aeffd640 10 monclient: dump: 2024-03-28T16:15:55.140 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:55.140 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:55.140 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:55.140 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:55.140 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:55.140 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:55.140 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:55.140 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:55.141 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:55.141 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:55.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.136+0000 7f97aeffd640 10 monclient: _finish_auth 0 2024-03-28T16:15:55.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.136+0000 7f97aeffd640 10 monclient: _check_auth_tickets 2024-03-28T16:15:55.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.136+0000 7f97aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:55.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.136+0000 7f97aeffd640 1 -- 172.21.15.7:0/2879071176 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b0003cd0 con 0x7f97b813ed70 2024-03-28T16:15:55.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.136+0000 7f97bcda4640 5 monclient: authenticate success, global_id 4316 2024-03-28T16:15:55.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.136+0000 7f97bcda4640 10 monclient: _renew_subs 2024-03-28T16:15:55.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.136+0000 7f97bcda4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:55.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.136+0000 7f97bcda4640 1 -- 172.21.15.7:0/2879071176 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b8144940 con 0x7f97b813ed70 2024-03-28T16:15:55.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.137+0000 7f97aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:55.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.137+0000 7f97aeffd640 1 -- 172.21.15.7:0/2879071176 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b000aa90 con 0x7f97b813ed70 2024-03-28T16:15:55.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.137+0000 7f97aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:55.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.137+0000 7f97aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-28T16:15:55.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.137+0000 7f97aeffd640 10 monclient: dump: 2024-03-28T16:15:55.142 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:55.142 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:55.142 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:55.142 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:55.142 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:55.142 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:55.142 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:55.142 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:55.142 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:55.142 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:55.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.137+0000 7f97b4d99640 4 set_mon_vals no callback set 2024-03-28T16:15:55.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.137+0000 7f97bcda4640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:55.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.137+0000 7f97bcda4640 10 monclient: _renew_subs 2024-03-28T16:15:55.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.137+0000 7f97bcda4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:55.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.137+0000 7f97bcda4640 1 -- 172.21.15.7:0/2879071176 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b81d79c0 con 0x7f97b813ed70 2024-03-28T16:15:55.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.137+0000 7f97bcda4640 1 librados: init done 2024-03-28T16:15:55.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.138+0000 7f97bcda4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:55.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.138+0000 7f97bcda4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:55.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.138+0000 7f97bcda4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2024-03-28T16:15:55.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.138+0000 7f97bcda4640 1 -- 172.21.15.7:0/2879071176 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97b81d6fa0 con 0x7f97b813ed70 2024-03-28T16:15:55.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.156+0000 7f97aeffd640 1 -- 172.21.15.7:0/2879071176 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f97b000abf0 con 0x7f97b813ed70 2024-03-28T16:15:55.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.156+0000 7f97aeffd640 1 --2- 172.21.15.7:0/2879071176 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f979c06c5e0 0x7f979c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:55.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.157+0000 7f97b659c640 1 --2- 172.21.15.7:0/2879071176 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f979c06c5e0 0x7f979c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:55.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.157+0000 7f97b659c640 10 monclient: get_auth_request con 0x7f979c06c5e0 auth_method 0 2024-03-28T16:15:55.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.157+0000 7f97b659c640 1 --2- 172.21.15.7:0/2879071176 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f979c06c5e0 0x7f979c06eaa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f97a0036000 tx=0x7f97a003c040 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:55.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.157+0000 7f97aeffd640 10 client.4316.objecter ms_handle_connect 0x7f979c06c5e0 2024-03-28T16:15:55.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.158+0000 7f97aeffd640 1 -- 172.21.15.7:0/2879071176 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f97b0058b30 con 0x7f97b813ed70 2024-03-28T16:15:55.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.158+0000 7f97aeffd640 10 client.4316.objecter ms_dispatch 0x7f97b8000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:55.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.158+0000 7f97aeffd640 3 client.4316.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:55.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.158+0000 7f97aeffd640 3 client.4316.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:55.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.158+0000 7f97aeffd640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-28T16:15:55.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.411+0000 7f97aeffd640 1 -- 172.21.15.7:0/2879071176 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f97b0054e70 con 0x7f97b813ed70 2024-03-28T16:15:55.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.411+0000 7f97aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:55.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.411+0000 7f97aeffd640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:55.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.593+0000 7f97bcda4640 1 -- 172.21.15.7:0/2879071176 --> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f97b81d6c10 con 0x7f979c06c5e0 2024-03-28T16:15:55.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.596+0000 7f97aeffd640 1 -- 172.21.15.7:0/2879071176 <== mgr.4162 v2:172.21.15.83:6800/34752 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47964 (secure 0 0 0) 0x7f97b81d6c10 con 0x7f979c06c5e0 2024-03-28T16:15:55.597 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:15:55.597 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-28T16:15:55.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.597+0000 7f97bcda4640 10 librados: watch_flush enter 2024-03-28T16:15:55.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.597+0000 7f97bcda4640 10 librados: watch_flush exit 2024-03-28T16:15:55.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.597+0000 7f97bcda4640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-28T16:15:55.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.597+0000 7f97bcda4640 10 client.4316.objecter successfully canceled tick 2024-03-28T16:15:55.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.597+0000 7f97bcda4640 1 -- 172.21.15.7:0/2879071176 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f979c06c5e0 msgr2=0x7f979c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:55.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.597+0000 7f97bcda4640 1 --2- 172.21.15.7:0/2879071176 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f979c06c5e0 0x7f979c06eaa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f97a0036000 tx=0x7f97a003c040 comp rx=0 tx=0).stop 2024-03-28T16:15:55.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.597+0000 7f97bcda4640 10 monclient: shutdown 2024-03-28T16:15:55.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.597+0000 7f97bcda4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:55.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.598+0000 7f97bcda4640 1 -- 172.21.15.7:0/2879071176 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f97b813ed70 msgr2=0x7f97b813f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:55.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.598+0000 7f97bcda4640 1 --2- 172.21.15.7:0/2879071176 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f97b813ed70 0x7f97b813f350 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f97b000c800 tx=0x7f97b000ccc0 comp rx=0 tx=0).stop 2024-03-28T16:15:55.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.598+0000 7f97bcda4640 1 -- 172.21.15.7:0/2879071176 shutdown_connections 2024-03-28T16:15:55.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.598+0000 7f97bcda4640 1 --2- 172.21.15.7:0/2879071176 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f979c06c5e0 0x7f979c06eaa0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:55.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.598+0000 7f97bcda4640 1 --2- 172.21.15.7:0/2879071176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97b8140d40 0x7f97b81d72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:55.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.598+0000 7f97bcda4640 1 --2- 172.21.15.7:0/2879071176 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f97b813ed70 0x7f97b813f350 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:55.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.598+0000 7f97bcda4640 1 --2- 172.21.15.7:0/2879071176 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97b8071920 0x7f97b8145450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:55.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.598+0000 7f97bcda4640 1 -- 172.21.15.7:0/2879071176 >> 172.21.15.7:0/2879071176 conn(0x7f97b80b3db0 msgr2=0x7f97b8139a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:55.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.598+0000 7f97bcda4640 1 -- 172.21.15.7:0/2879071176 shutdown_connections 2024-03-28T16:15:55.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.598+0000 7f97bcda4640 1 -- 172.21.15.7:0/2879071176 wait complete. 2024-03-28T16:15:55.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.598+0000 7f97bcda4640 1 librados: shutdown 2024-03-28T16:15:55.609 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-03-28T16:15:55.042059+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219196,"kb_used_data":3024,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755524,"statfs":{"total":773094113280,"available":772869656576,"internally_reserved":0,"allocated":3096576,"data_stored":1579744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001241"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-03-28T16:15:49.154143+0000","last_change":"2024-03-28T16:15:49.142733+0000","last_active":"2024-03-28T16:15:49.154143+0000","last_peered":"2024-03-28T16:15:49.154143+0000","last_clean":"2024-03-28T16:15:49.154143+0000","last_became_active":"2024-03-28T16:15:49.135276+0000","last_became_peered":"2024-03-28T16:15:49.135276+0000","last_unstale":"2024-03-28T16:15:49.154143+0000","last_undegraded":"2024-03-28T16:15:49.154143+0000","last_fullsized":"2024-03-28T16:15:49.154143+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_clean_scrub_stamp":"2024-03-28T16:15:46.495373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:12:40.798563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-03-28T16:15:49.196738+0000","last_change":"2024-03-28T16:15:49.179203+0000","last_active":"2024-03-28T16:15:49.196738+0000","last_peered":"2024-03-28T16:15:49.196738+0000","last_clean":"2024-03-28T16:15:49.196738+0000","last_became_active":"2024-03-28T16:15:49.153269+0000","last_became_peered":"2024-03-28T16:15:49.153269+0000","last_unstale":"2024-03-28T16:15:49.196738+0000","last_undegraded":"2024-03-28T16:15:49.196738+0000","last_fullsized":"2024-03-28T16:15:49.196738+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_clean_scrub_stamp":"2024-03-28T16:15:46.495373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:16.114251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-03-28T16:15:48.613882+0000","last_change":"2024-03-28T16:15:48.601259+0000","last_active":"2024-03-28T16:15:48.613882+0000","last_peered":"2024-03-28T16:15:48.613882+0000","last_clean":"2024-03-28T16:15:48.613882+0000","last_became_active":"2024-03-28T16:15:48.596033+0000","last_became_peered":"2024-03-28T16:15:48.596033+0000","last_unstale":"2024-03-28T16:15:48.613882+0000","last_undegraded":"2024-03-28T16:15:48.613882+0000","last_fullsized":"2024-03-28T16:15:48.613882+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_clean_scrub_stamp":"2024-03-28T16:15:46.495373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:39:38.041200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-03-28T16:15:50.047529+0000","last_change":"2024-03-28T16:15:48.819449+0000","last_active":"2024-03-28T16:15:50.047529+0000","last_peered":"2024-03-28T16:15:50.047529+0000","last_clean":"2024-03-28T16:15:50.047529+0000","last_became_active":"2024-03-28T16:15:48.790674+0000","last_became_peered":"2024-03-28T16:15:48.790674+0000","last_unstale":"2024-03-28T16:15:50.047529+0000","last_undegraded":"2024-03-28T16:15:50.047529+0000","last_fullsized":"2024-03-28T16:15:50.047529+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_clean_scrub_stamp":"2024-03-28T16:15:46.495373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:33:30.273205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-03-28T16:15:48.783171+0000","last_change":"2024-03-28T16:15:48.773840+0000","last_active":"2024-03-28T16:15:48.783171+0000","last_peered":"2024-03-28T16:15:48.783171+0000","last_clean":"2024-03-28T16:15:48.783171+0000","last_became_active":"2024-03-28T16:15:48.764041+0000","last_became_peered":"2024-03-28T16:15:48.764041+0000","last_unstale":"2024-03-28T16:15:48.783171+0000","last_undegraded":"2024-03-28T16:15:48.783171+0000","last_fullsized":"2024-03-28T16:15:48.783171+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_clean_scrub_stamp":"2024-03-28T16:15:46.495373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:51:37.011076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-03-28T16:15:50.049241+0000","last_change":"2024-03-28T16:15:48.502499+0000","last_active":"2024-03-28T16:15:50.049241+0000","last_peered":"2024-03-28T16:15:50.049241+0000","last_clean":"2024-03-28T16:15:50.049241+0000","last_became_active":"2024-03-28T16:15:48.474531+0000","last_became_peered":"2024-03-28T16:15:48.474531+0000","last_unstale":"2024-03-28T16:15:50.049241+0000","last_undegraded":"2024-03-28T16:15:50.049241+0000","last_fullsized":"2024-03-28T16:15:50.049241+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_clean_scrub_stamp":"2024-03-28T16:15:46.495373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:11:08.194988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-03-28T16:15:48.611809+0000","last_change":"2024-03-28T16:15:48.549497+0000","last_active":"2024-03-28T16:15:48.611809+0000","last_peered":"2024-03-28T16:15:48.611809+0000","last_clean":"2024-03-28T16:15:48.611809+0000","last_became_active":"2024-03-28T16:15:48.521388+0000","last_became_peered":"2024-03-28T16:15:48.521388+0000","last_unstale":"2024-03-28T16:15:48.611809+0000","last_undegraded":"2024-03-28T16:15:48.611809+0000","last_fullsized":"2024-03-28T16:15:48.611809+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_clean_scrub_stamp":"2024-03-28T16:15:46.495373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:43:06.462972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-03-28T16:15:48.853506+0000","last_change":"2024-03-28T16:15:48.847000+0000","last_active":"2024-03-28T16:15:48.853506+0000","last_peered":"2024-03-28T16:15:48.853506+0000","last_clean":"2024-03-28T16:15:48.853506+0000","last_became_active":"2024-03-28T16:15:48.841691+0000","last_became_peered":"2024-03-28T16:15:48.841691+0000","last_unstale":"2024-03-28T16:15:48.853506+0000","last_undegraded":"2024-03-28T16:15:48.853506+0000","last_fullsized":"2024-03-28T16:15:48.853506+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:15:46.495373+0000","last_clean_scrub_stamp":"2024-03-28T16:15:46.495373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:09:19.425223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":133,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-03-28T16:15:48.188781+0000","last_change":"2024-03-28T16:15:45.207007+0000","last_active":"2024-03-28T16:15:48.188781+0000","last_peered":"2024-03-28T16:15:48.188781+0000","last_clean":"2024-03-28T16:15:48.188781+0000","last_became_active":"2024-03-28T16:15:44.645475+0000","last_became_peered":"2024-03-28T16:15:44.645475+0000","last_unstale":"2024-03-28T16:15:48.188781+0000","last_undegraded":"2024-03-28T16:15:48.188781+0000","last_fullsized":"2024-03-28T16:15:48.188781+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T16:15:31.785347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T16:15:31.785347+0000","last_clean_scrub_stamp":"2024-03-28T16:15:31.785347+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:19:25.283430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":23,"seq":98784247816,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":688128,"data_stored":495848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4489999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.4949999999999992}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.786}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.372}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.223000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.813000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.807}]}]},{"osd":6,"up_from":21,"seq":90194313228,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":98008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.1600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2839999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.827}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.919}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.67}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.206}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.287000000000001}]}]},{"osd":3,"up_from":17,"seq":73014444041,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":98008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.7960000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.844999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.713000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.756}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.201000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.061999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.819000000000001}]}]},{"osd":2,"up_from":17,"seq":73014444041,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":98008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.077}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.219999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.037000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.352}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.807}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.339}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.2530000000000001}]}]},{"osd":1,"up_from":17,"seq":73014444039,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":688128,"data_stored":495848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.605}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8140000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":61.380000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.196}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.217000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.089}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.711}]}]},{"osd":7,"up_from":15,"seq":64424509448,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":98008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.137}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0860000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.57}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":202.93299999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.0540000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.91}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.513999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":98008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.696999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0719999999999992}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.706}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.725999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.757999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":30.998999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":32.948}]}]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":98008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.8049999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.065999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.693000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.587}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.516999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.710000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.434000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T16:15:55.610 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-28T16:15:55.610 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-28T16:15:55.610 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-28T16:15:55.610 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-28T16:15:55.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.709+0000 7f265a2e7640 10 monclient: get_monmap_and_config 2024-03-28T16:15:55.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.709+0000 7f265a2e7640 10 monclient: build_initial_monmap 2024-03-28T16:15:55.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.709+0000 7f265a2e7640 10 monclient: monmap: 2024-03-28T16:15:55.711 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:55.711 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:55.711 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:15:55.711257+0000 2024-03-28T16:15:55.711 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:15:55.711257+0000 2024-03-28T16:15:55.711 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:55.711 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:55.711 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T16:15:55.711 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T16:15:55.711 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T16:15:55.711 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T16:15:55.711 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T16:15:55.711 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:55.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.710+0000 7f265a2e7640 1 Processor -- start 2024-03-28T16:15:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.710+0000 7f265a2e7640 1 -- start start 2024-03-28T16:15:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.710+0000 7f265a2e7640 10 monclient: init 2024-03-28T16:15:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.710+0000 7f265a2e7640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.711+0000 7f265a2e7640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-03-28T16:15:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.711+0000 7f265a2e7640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f26540717f0 0x7f265413ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.711+0000 7f265a2e7640 10 monclient(hunting): picked mon.noname-a con 0x7f26540717f0 addr v2:172.21.15.7:3300/0 2024-03-28T16:15:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.711+0000 7f265a2e7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2654071bd0 addr v1:172.21.15.83:6789/0 2024-03-28T16:15:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.711+0000 7f265a2e7640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f26540b0f50 0x7f265414bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.711+0000 7f265a2e7640 10 monclient(hunting): picked mon.noname-c con 0x7f26540b0f50 addr v2:172.21.15.83:3300/0 2024-03-28T16:15:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.711+0000 7f265a2e7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2654141710 con 0x7f2654071bd0 2024-03-28T16:15:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.711+0000 7f265a2e7640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.711+0000 7f265a2e7640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26540b02b0 con 0x7f26540717f0 2024-03-28T16:15:55.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.711+0000 7f265a2e7640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:55.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.711+0000 7f265a2e7640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26540b0590 con 0x7f26540b0f50 2024-03-28T16:15:55.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.711+0000 7f265a2e7640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:55.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.711+0000 7f265a2e7640 10 monclient(hunting): authenticate will time out at 1422.852942s 2024-03-28T16:15:55.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.711+0000 7f2653fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f26540717f0 0x7f265413ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:55.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.711+0000 7f2658884640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f26540b0f50 0x7f265414bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:55.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.712+0000 7f2653fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f26540717f0 0x7f265413ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35464/0 (socket says 172.21.15.7:35464) 2024-03-28T16:15:55.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.712+0000 7f2653fff640 1 -- 172.21.15.7:0/1189339308 learned_addr learned my addr 172.21.15.7:0/1189339308 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:55.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.712+0000 7f2653fff640 10 monclient(hunting): get_auth_request con 0x7f26540717f0 auth_method 0 2024-03-28T16:15:55.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.712+0000 7f2653fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:55.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.712+0000 7f2653fff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:55.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.712+0000 7f2653fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:55.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.712+0000 7f2658884640 10 monclient(hunting): get_auth_request con 0x7f26540b0f50 auth_method 0 2024-03-28T16:15:55.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.712+0000 7f2658884640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:55.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.712+0000 7f2658884640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:55.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.712+0000 7f2658884640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:55.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.712+0000 7f2653fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:55.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.712+0000 7f2653fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:55.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.712+0000 7f2653fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:55.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.712+0000 7f2658884640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:55.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.712+0000 7f2658884640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:55.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.712+0000 7f2658884640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:55.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.714+0000 7f2653fff640 10 monclient(hunting): handle_auth_done global_id 4377 payload 274 2024-03-28T16:15:55.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.714+0000 7f2653fff640 1 -- 172.21.15.7:0/1189339308 >> v2:172.21.15.83:3300/0 conn(0x7f26540b0f50 msgr2=0x7f265414bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:55.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.714+0000 7f2653fff640 1 --2- 172.21.15.7:0/1189339308 >> v2:172.21.15.83:3300/0 conn(0x7f26540b0f50 0x7f265414bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:55.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.714+0000 7f2653fff640 1 -- 172.21.15.7:0/1189339308 >> v1:172.21.15.83:6789/0 conn(0x7f2654071bd0 legacy=0x7f26540ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:15:55.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.714+0000 7f2653fff640 10 monclient: _finish_hunting 0 2024-03-28T16:15:55.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.714+0000 7f2653fff640 1 monclient: found mon.noname-a 2024-03-28T16:15:55.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.714+0000 7f2653fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:55.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.714+0000 7f2653fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2024-03-28T16:15:55.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.714+0000 7f2653fff640 1 -- 172.21.15.7:0/1189339308 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26540b13f0 con 0x7f26540717f0 2024-03-28T16:15:55.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.716+0000 7f2653fff640 1 --2- 172.21.15.7:0/1189339308 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26540717f0 0x7f265413ece0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f26540af930 tx=0x7f264802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de5f57a6916d3fdd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:55.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.720+0000 7f2652ffd640 1 -- 172.21.15.7:0/1189339308 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2648031870 con 0x7f26540717f0 2024-03-28T16:15:55.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.720+0000 7f2652ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:55.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.720+0000 7f2652ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-28T16:15:55.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.720+0000 7f2652ffd640 10 monclient: dump: 2024-03-28T16:15:55.721 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:55.721 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:55.721 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:55.722 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:55.722 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:55.722 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:55.722 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:55.722 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:55.722 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:55.722 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:55.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.720+0000 7f2652ffd640 10 monclient: _finish_auth 0 2024-03-28T16:15:55.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.720+0000 7f2652ffd640 10 monclient: _check_auth_tickets 2024-03-28T16:15:55.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.720+0000 7f2652ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:55.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.720+0000 7f265a2e7640 5 monclient: authenticate success, global_id 4377 2024-03-28T16:15:55.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.720+0000 7f265a2e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-28T16:15:55.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.721+0000 7f2652ffd640 1 -- 172.21.15.7:0/1189339308 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26480319d0 con 0x7f26540717f0 2024-03-28T16:15:55.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.721+0000 7f2652ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:55.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.721+0000 7f2652ffd640 1 -- 172.21.15.7:0/1189339308 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2648031d60 con 0x7f26540717f0 2024-03-28T16:15:55.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.721+0000 7f2652ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:55.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.721+0000 7f2652ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-28T16:15:55.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.721+0000 7f2652ffd640 10 monclient: dump: 2024-03-28T16:15:55.724 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:55.724 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:55.724 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:55.724 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:55.724 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:55.724 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:55.724 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:55.724 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:55.724 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:55.724 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:55.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.721+0000 7f265a2e7640 10 monclient: get_monmap_and_config success 2024-03-28T16:15:55.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.721+0000 7f265a2e7640 4 set_mon_vals no callback set 2024-03-28T16:15:55.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.722+0000 7f265a2e7640 10 monclient: shutdown 2024-03-28T16:15:55.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.722+0000 7f265a2e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:55.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.722+0000 7f265a2e7640 1 -- 172.21.15.7:0/1189339308 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26540717f0 msgr2=0x7f265413ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:55.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.722+0000 7f265a2e7640 1 --2- 172.21.15.7:0/1189339308 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26540717f0 0x7f265413ece0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f26540af930 tx=0x7f264802b910 comp rx=0 tx=0).stop 2024-03-28T16:15:55.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.722+0000 7f265a2e7640 1 -- 172.21.15.7:0/1189339308 shutdown_connections 2024-03-28T16:15:55.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.722+0000 7f265a2e7640 1 --2- 172.21.15.7:0/1189339308 >> v2:172.21.15.83:3300/0 conn(0x7f26540b0f50 0x7f265414bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:55.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.722+0000 7f265a2e7640 1 --2- 172.21.15.7:0/1189339308 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26540717f0 0x7f265413ece0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:55.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.722+0000 7f265a2e7640 1 -- 172.21.15.7:0/1189339308 >> 172.21.15.7:0/1189339308 conn(0x7f26540b7320 msgr2=0x7f26540b7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:55.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.722+0000 7f265a2e7640 1 -- 172.21.15.7:0/1189339308 shutdown_connections 2024-03-28T16:15:55.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.722+0000 7f265a2e7640 1 -- 172.21.15.7:0/1189339308 wait complete. 2024-03-28T16:15:55.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.723+0000 7f265a2e7640 10 monclient: build_initial_monmap 2024-03-28T16:15:55.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.723+0000 7f265a2e7640 10 monclient: monmap: 2024-03-28T16:15:55.729 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T16:15:55.729 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:15:55.729 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-28T16:15:55.729 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-28T16:15:55.729 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T16:15:55.729 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:55.729 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:15:55.729 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:15:55.729 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:15:55.729 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:55.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.723+0000 7f265a2e7640 1 librados: starting msgr at 2024-03-28T16:15:55.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.723+0000 7f265a2e7640 1 librados: starting objecter 2024-03-28T16:15:55.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.723+0000 7f265a2e7640 1 Processor -- start 2024-03-28T16:15:55.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.724+0000 7f265a2e7640 1 -- start start 2024-03-28T16:15:55.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.724+0000 7f265a2e7640 1 librados: setting wanted keys 2024-03-28T16:15:55.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.724+0000 7f265a2e7640 1 librados: calling monclient init 2024-03-28T16:15:55.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.724+0000 7f265a2e7640 10 monclient: init 2024-03-28T16:15:55.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.724+0000 7f265a2e7640 10 monclient: _reopen_session rank -1 2024-03-28T16:15:55.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.724+0000 7f265a2e7640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-28T16:15:55.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.724+0000 7f265a2e7640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26540717f0 0x7f26541c9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:55.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.724+0000 7f265a2e7640 10 monclient(hunting): picked mon.noname-a con 0x7f26540717f0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:15:55.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.724+0000 7f265a2e7640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f26540b0f50 0x7f26541c7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:55.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.724+0000 7f265a2e7640 10 monclient(hunting): picked mon.noname-c con 0x7f26540b0f50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:15:55.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.724+0000 7f265a2e7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26541c7d60 0x7f26541cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:55.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.724+0000 7f265a2e7640 10 monclient(hunting): picked mon.noname-b con 0x7f26541c7d60 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:15:55.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.724+0000 7f265a2e7640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:55.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.724+0000 7f265a2e7640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265413d080 con 0x7f26540717f0 2024-03-28T16:15:55.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.724+0000 7f265a2e7640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:55.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.724+0000 7f265a2e7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26541cacb0 con 0x7f26541c7d60 2024-03-28T16:15:55.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.724+0000 7f265a2e7640 10 monclient(hunting): start opening mon connection 2024-03-28T16:15:55.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.724+0000 7f265a2e7640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26541c81d0 con 0x7f26540b0f50 2024-03-28T16:15:55.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.724+0000 7f265a2e7640 10 monclient(hunting): _renew_subs 2024-03-28T16:15:55.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.724+0000 7f265a2e7640 10 monclient(hunting): authenticate will time out at 1422.866450s 2024-03-28T16:15:55.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f2653fff640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26540717f0 0x7f26541c9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:55.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f2658884640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26541c7d60 0x7f26541cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:55.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f26537fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f26540b0f50 0x7f26541c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:55.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f2658884640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26541c7d60 0x7f26541cdc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:38802/0 (socket says 172.21.15.7:38802) 2024-03-28T16:15:55.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f2658884640 1 -- 172.21.15.7:0/949366771 learned_addr learned my addr 172.21.15.7:0/949366771 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:15:55.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f2658884640 10 monclient(hunting): get_auth_request con 0x7f26541c7d60 auth_method 0 2024-03-28T16:15:55.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f2658884640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:55.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f2658884640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:55.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f2658884640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:55.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f26537fe640 10 monclient(hunting): get_auth_request con 0x7f26540b0f50 auth_method 0 2024-03-28T16:15:55.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f26537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:55.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f26537fe640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:55.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f26537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:55.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f2653fff640 10 monclient(hunting): get_auth_request con 0x7f26540717f0 auth_method 0 2024-03-28T16:15:55.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f2653fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:15:55.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f2653fff640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:15:55.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f2653fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:15:55.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f26537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:55.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f26537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:55.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f26537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:55.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f2658884640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:55.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f2658884640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:55.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.725+0000 7f2658884640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:55.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.726+0000 7f2653fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:15:55.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.726+0000 7f2653fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:15:55.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.726+0000 7f2653fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:15:55.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.728+0000 7f2658884640 10 monclient(hunting): handle_auth_done global_id 4423 payload 824 2024-03-28T16:15:55.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.728+0000 7f2658884640 1 -- 172.21.15.7:0/949366771 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f26540b0f50 msgr2=0x7f26541c7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:55.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.728+0000 7f2658884640 1 --2- 172.21.15.7:0/949366771 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f26540b0f50 0x7f26541c7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:55.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.728+0000 7f2658884640 1 -- 172.21.15.7:0/949366771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26540717f0 msgr2=0x7f26541c9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:55.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.728+0000 7f2658884640 1 --2- 172.21.15.7:0/949366771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26540717f0 0x7f26541c9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:55.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.728+0000 7f2658884640 10 monclient: _finish_hunting 0 2024-03-28T16:15:55.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.728+0000 7f2658884640 1 monclient: found mon.noname-b 2024-03-28T16:15:55.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.728+0000 7f2658884640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:15:55.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.728+0000 7f2658884640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:15:55.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.728+0000 7f2658884640 1 -- 172.21.15.7:0/949366771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26480094d0 con 0x7f26541c7d60 2024-03-28T16:15:55.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.731+0000 7f2658884640 1 --2- 172.21.15.7:0/949366771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26541c7d60 0x7f26541cdc60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f264c009d30 tx=0x7f264c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:55.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.731+0000 7f2650ff9640 10 client.?.objecter ms_handle_connect 0x7f26541c7d60 2024-03-28T16:15:55.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.731+0000 7f2650ff9640 10 client.?.objecter resend_mon_ops 2024-03-28T16:15:55.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f2650ff9640 1 -- 172.21.15.7:0/949366771 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c009ba0 con 0x7f26541c7d60 2024-03-28T16:15:55.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f2650ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:55.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f2650ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:15:55.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f2650ff9640 10 monclient: dump: 2024-03-28T16:15:55.737 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:55.737 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:55.737 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:55.737 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:55.738 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:55.738 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:55.738 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:55.738 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:55.738 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:55.738 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:55.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f2650ff9640 10 monclient: _finish_auth 0 2024-03-28T16:15:55.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f2650ff9640 10 monclient: _check_auth_tickets 2024-03-28T16:15:55.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f2650ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:15:55.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f2650ff9640 1 -- 172.21.15.7:0/949366771 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f264c00f040 con 0x7f26541c7d60 2024-03-28T16:15:55.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f2650ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:15:55.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f2650ff9640 1 -- 172.21.15.7:0/949366771 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c0146c0 con 0x7f26541c7d60 2024-03-28T16:15:55.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f2650ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:15:55.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f2650ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:15:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f2650ff9640 10 monclient: dump: 2024-03-28T16:15:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-28T16:15:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:15:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:15:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 18 (reef) 2024-03-28T16:15:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T16:15:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:15:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:15:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:15:55.739 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T16:15:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f265a2e7640 5 monclient: authenticate success, global_id 4423 2024-03-28T16:15:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f265a2e7640 10 monclient: _renew_subs 2024-03-28T16:15:55.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f265a2e7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:55.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f265a2e7640 1 -- 172.21.15.7:0/949366771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26541cc370 con 0x7f26541c7d60 2024-03-28T16:15:55.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f26527fc640 4 set_mon_vals no callback set 2024-03-28T16:15:55.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f265a2e7640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:15:55.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f265a2e7640 10 monclient: _renew_subs 2024-03-28T16:15:55.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f265a2e7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:55.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f265a2e7640 1 -- 172.21.15.7:0/949366771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26541c9190 con 0x7f26541c7d60 2024-03-28T16:15:55.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.735+0000 7f265a2e7640 1 librados: init done 2024-03-28T16:15:55.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.736+0000 7f265a2e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:55.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.736+0000 7f265a2e7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:55.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.736+0000 7f265a2e7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:55.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.736+0000 7f265a2e7640 1 -- 172.21.15.7:0/949366771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26541cae60 con 0x7f26541c7d60 2024-03-28T16:15:55.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.981+0000 7f2650ff9640 1 -- 172.21.15.7:0/949366771 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f264c014820 con 0x7f26541c7d60 2024-03-28T16:15:55.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.981+0000 7f2650ff9640 1 --2- 172.21.15.7:0/949366771 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f264006c5c0 0x7f264006ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:15:55.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.982+0000 7f2653fff640 1 --2- 172.21.15.7:0/949366771 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f264006c5c0 0x7f264006ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:15:55.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.982+0000 7f2653fff640 10 monclient: get_auth_request con 0x7f264006c5c0 auth_method 0 2024-03-28T16:15:55.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.982+0000 7f2653fff640 1 --2- 172.21.15.7:0/949366771 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f264006c5c0 0x7f264006ea80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2648036000 tx=0x7f264803c040 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:15:55.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.982+0000 7f2650ff9640 10 client.4423.objecter ms_handle_connect 0x7f264006c5c0 2024-03-28T16:15:55.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.984+0000 7f2650ff9640 1 -- 172.21.15.7:0/949366771 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f264c059310 con 0x7f26541c7d60 2024-03-28T16:15:55.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.984+0000 7f2650ff9640 10 client.4423.objecter ms_dispatch 0x7f2654000b90 osd_map(28..28 src has 1..28) v4 2024-03-28T16:15:55.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.984+0000 7f2650ff9640 3 client.4423.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:15:55.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.984+0000 7f2650ff9640 3 client.4423.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:15:55.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.984+0000 7f2650ff9640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-28T16:15:55.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.997+0000 7f2650ff9640 1 -- 172.21.15.7:0/949366771 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f264c055650 con 0x7f26541c7d60 2024-03-28T16:15:55.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.997+0000 7f2650ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-28T16:15:55.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:55.997+0000 7f2650ff9640 10 monclient: _finish_command 1 = system:0 2024-03-28T16:15:56.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.241+0000 7f265a2e7640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-03-28T16:15:56.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.241+0000 7f265a2e7640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-03-28T16:15:56.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.241+0000 7f265a2e7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:15:56.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.241+0000 7f265a2e7640 1 -- 172.21.15.7:0/949366771 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f26541cae60 con 0x7f26541c7d60 2024-03-28T16:15:56.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.247+0000 7f2650ff9640 1 -- 172.21.15.7:0/949366771 <== mon.1 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f26541cae60 con 0x7f26541c7d60 2024-03-28T16:15:56.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.247+0000 7f2650ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-03-28T16:15:56.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.247+0000 7f2650ff9640 10 monclient: _finish_command 2 = system:0 2024-03-28T16:15:56.248 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-28T16:15:56.248 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-28T16:15:56.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.248+0000 7f265a2e7640 10 librados: watch_flush enter 2024-03-28T16:15:56.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.248+0000 7f265a2e7640 10 librados: watch_flush exit 2024-03-28T16:15:56.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.248+0000 7f265a2e7640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-28T16:15:56.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.248+0000 7f265a2e7640 10 client.4423.objecter successfully canceled tick 2024-03-28T16:15:56.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.249+0000 7f265a2e7640 1 -- 172.21.15.7:0/949366771 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f264006c5c0 msgr2=0x7f264006ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:56.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.249+0000 7f265a2e7640 1 --2- 172.21.15.7:0/949366771 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f264006c5c0 0x7f264006ea80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2648036000 tx=0x7f264803c040 comp rx=0 tx=0).stop 2024-03-28T16:15:56.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.249+0000 7f265a2e7640 10 monclient: shutdown 2024-03-28T16:15:56.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.249+0000 7f265a2e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:15:56.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.249+0000 7f265a2e7640 1 -- 172.21.15.7:0/949366771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26541c7d60 msgr2=0x7f26541cdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:15:56.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.249+0000 7f265a2e7640 1 --2- 172.21.15.7:0/949366771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26541c7d60 0x7f26541cdc60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f264c009d30 tx=0x7f264c00c490 comp rx=0 tx=0).stop 2024-03-28T16:15:56.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.249+0000 7f265a2e7640 1 -- 172.21.15.7:0/949366771 shutdown_connections 2024-03-28T16:15:56.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.249+0000 7f265a2e7640 1 --2- 172.21.15.7:0/949366771 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7f264006c5c0 0x7f264006ea80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:56.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.249+0000 7f265a2e7640 1 --2- 172.21.15.7:0/949366771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26541c7d60 0x7f26541cdc60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:56.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.249+0000 7f265a2e7640 1 --2- 172.21.15.7:0/949366771 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f26540b0f50 0x7f26541c7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:56.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.249+0000 7f265a2e7640 1 --2- 172.21.15.7:0/949366771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26540717f0 0x7f26541c9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:15:56.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.249+0000 7f265a2e7640 1 -- 172.21.15.7:0/949366771 >> 172.21.15.7:0/949366771 conn(0x7f26540b7320 msgr2=0x7f26540b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:15:56.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.249+0000 7f265a2e7640 1 -- 172.21.15.7:0/949366771 shutdown_connections 2024-03-28T16:15:56.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.249+0000 7f265a2e7640 1 -- 172.21.15.7:0/949366771 wait complete. 2024-03-28T16:15:56.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T16:15:56.249+0000 7f265a2e7640 1 librados: shutdown 2024-03-28T16:15:56.260 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-28T16:15:56.262 INFO:teuthology.run_tasks:Running task workunit... 2024-03-28T16:15:56.277 INFO:tasks.workunit:Pulling workunits from ref e58b9979847f029f7ab1514b5f5f99b9f03d97f6 2024-03-28T16:15:56.277 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-28T16:15:56.278 DEBUG:teuthology.orchestra.run.smithi007:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-28T16:15:56.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T16:15:56.293 INFO:teuthology.orchestra.run.smithi007.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-28T16:15:56.293 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-28T16:15:56.348 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-28T16:15:56.348 DEBUG:teuthology.orchestra.run.smithi007:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-28T16:15:56.403 INFO:tasks.workunit:timeout=6h 2024-03-28T16:15:56.403 INFO:tasks.workunit:cleanup=True 2024-03-28T16:15:56.403 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e58b9979847f029f7ab1514b5f5f99b9f03d97f6 2024-03-28T16:15:56.458 INFO:tasks.workunit.client.0.smithi007.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-28T16:16:06.887 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:39.459 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:06.889 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:01:39.444 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:06.890 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:39.463 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:06.892 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:01:39.449 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:06.928 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:01:39.484 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:06.929 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:39.502 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:06.941 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:39.514 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:06.942 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:39.515 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:06.979 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:39.552 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:26.944 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:59.516 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:26.949 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:59.522 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:26.965 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:59.537 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:26.998 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:01:59.554 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:26.999 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:01:59.556 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:27.012 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:01:59.569 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:27.028 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:59.601 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:27.033 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:59.605 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:27.050 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:59.622 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:45.293 INFO:tasks.workunit.client.0.smithi007.stderr:Updating files: 99% (11993/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-03-28T16:16:45.880 INFO:tasks.workunit.client.0.smithi007.stderr:Note: switching to 'e58b9979847f029f7ab1514b5f5f99b9f03d97f6'. 2024-03-28T16:16:45.881 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-03-28T16:16:45.881 INFO:tasks.workunit.client.0.smithi007.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-28T16:16:45.881 INFO:tasks.workunit.client.0.smithi007.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-28T16:16:45.881 INFO:tasks.workunit.client.0.smithi007.stderr:state without impacting any branches by switching back to a branch. 2024-03-28T16:16:45.881 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-03-28T16:16:45.881 INFO:tasks.workunit.client.0.smithi007.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-28T16:16:45.881 INFO:tasks.workunit.client.0.smithi007.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-28T16:16:45.881 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-03-28T16:16:45.881 INFO:tasks.workunit.client.0.smithi007.stderr: git switch -c 2024-03-28T16:16:45.881 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-03-28T16:16:45.881 INFO:tasks.workunit.client.0.smithi007.stderr:Or undo this operation with: 2024-03-28T16:16:45.881 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-03-28T16:16:45.881 INFO:tasks.workunit.client.0.smithi007.stderr: git switch - 2024-03-28T16:16:45.882 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-03-28T16:16:45.882 INFO:tasks.workunit.client.0.smithi007.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-28T16:16:45.882 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-03-28T16:16:45.882 INFO:tasks.workunit.client.0.smithi007.stderr:HEAD is now at e58b9979847 Merge branch 'wip-65003-reef' of https://github.com/cbodley/ceph into wip-yuri4-testing-2024-03-27-1023-reef 2024-03-28T16:16:45.886 DEBUG:teuthology.orchestra.run.smithi007:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-28T16:16:46.017 INFO:tasks.workunit.client.0.smithi007.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-28T16:16:46.019 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T16:16:46.020 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-28T16:16:46.678 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-28T16:16:46.755 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-28T16:16:46.793 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T16:16:46.795 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T16:16:46.795 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-28T16:16:46.834 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T16:16:46.837 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-28T16:16:46.837 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-28T16:16:46.851 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-03-28T16:16:46.852 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-03-28T16:16:46.853 DEBUG:teuthology.orchestra.run.smithi007:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e58b9979847f029f7ab1514b5f5f99b9f03d97f6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-03-28T16:16:46.911 INFO:tasks.workunit.client.0.smithi007.stderr:+ parallel=1 2024-03-28T16:16:46.911 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' '' = --serial ']' 2024-03-28T16:16:46.911 INFO:tasks.workunit.client.0.smithi007.stderr:+ color= 2024-03-28T16:16:46.911 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -t 1 ']' 2024-03-28T16:16:46.911 INFO:tasks.workunit.client.0.smithi007.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-03-28T16:16:46.911 INFO:tasks.workunit.client.0.smithi007.stderr:+ declare -A pids 2024-03-28T16:16:46.911 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:46.911 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:46.911 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_aio 2024-03-28T16:16:46.912 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_aio' 2024-03-28T16:16:46.912 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_aio 2024-03-28T16:16:46.912 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:46.912 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_aio 2024-03-28T16:16:46.912 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40074 2024-03-28T16:16:46.912 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_aio on pid 40074' 2024-03-28T16:16:46.913 INFO:tasks.workunit.client.0.smithi007.stdout:test api_aio on pid 40074 2024-03-28T16:16:46.916 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40074 2024-03-28T16:16:46.916 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-03-28T16:16:46.916 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:46.916 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:46.916 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_aio_pp 2024-03-28T16:16:46.916 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_aio_pp' 2024-03-28T16:16:46.916 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_aio_pp 2024-03-28T16:16:46.916 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:46.916 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:46.916 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:46.917 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:46.917 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:46.917 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.917 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:46.917 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:46.917 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:46.917 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:46.917 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.917 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:46.917 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.917 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:46.917 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:46.917 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.917 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:46.917 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.918 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:46.918 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:46.918 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:46.918 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.918 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:46.918 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.918 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:46.918 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:46.918 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_aio_pp 2024-03-28T16:16:46.918 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40081 2024-03-28T16:16:46.918 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_aio_pp on pid 40081' 2024-03-28T16:16:46.918 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40081 2024-03-28T16:16:46.919 INFO:tasks.workunit.client.0.smithi007.stdout:test api_aio_pp on pid 40081 2024-03-28T16:16:46.923 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:46.923 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:46.923 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-03-28T16:16:46.923 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_io 2024-03-28T16:16:46.923 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_io' 2024-03-28T16:16:46.923 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_io 2024-03-28T16:16:46.923 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:46.923 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:46.923 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:46.923 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:46.924 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:46.924 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.924 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:46.924 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:46.924 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:46.924 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:46.924 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:46.924 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.924 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:46.924 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.924 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:46.924 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:46.924 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:46.924 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:46.924 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.925 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:46.925 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:46.925 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.925 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:46.925 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:46.925 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:46.925 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.925 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:46.925 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.925 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.925 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:46.925 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:46.925 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.925 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:46.925 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:46.926 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:46.926 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.926 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:46.926 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.926 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:46.926 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:46.926 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:46.926 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.926 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:46.926 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.926 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:46.926 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:46.926 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_io 2024-03-28T16:16:46.926 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40090 2024-03-28T16:16:46.927 INFO:tasks.workunit.client.0.smithi007.stdout:test api_io on pid 40090 2024-03-28T16:16:46.931 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_io on pid 40090' 2024-03-28T16:16:46.931 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40090 2024-03-28T16:16:46.932 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:46.932 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:46.932 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-03-28T16:16:46.932 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_io_pp 2024-03-28T16:16:46.932 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_io_pp' 2024-03-28T16:16:46.932 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_io_pp 2024-03-28T16:16:46.932 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:46.932 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:46.932 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:46.932 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:46.932 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:46.932 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.932 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:46.932 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:46.932 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:46.933 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:46.933 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:46.933 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.933 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:46.933 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:46.933 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.933 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:46.933 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:46.933 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.933 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:46.933 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.933 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:46.933 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:46.933 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:46.933 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:46.934 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.934 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:46.934 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.934 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:46.934 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:46.934 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:46.934 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.934 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:46.934 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.934 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:46.934 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:46.934 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:46.934 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:46.934 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:46.934 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:46.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:46.935 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:46.935 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:46.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:46.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:46.935 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:46.935 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:46.935 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_io_pp 2024-03-28T16:16:46.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:46.935 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.936 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:46.936 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:46.936 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40101 2024-03-28T16:16:46.936 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_io_pp on pid 40101' 2024-03-28T16:16:46.936 INFO:tasks.workunit.client.0.smithi007.stdout:test api_io_pp on pid 40101 2024-03-28T16:16:46.942 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40101 2024-03-28T16:16:46.942 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:46.942 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:46.942 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-03-28T16:16:46.942 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_asio 2024-03-28T16:16:46.942 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_asio' 2024-03-28T16:16:46.942 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_asio 2024-03-28T16:16:46.942 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:46.942 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:46.943 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:46.943 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:46.943 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:46.943 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:46.943 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:46.943 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:46.943 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.943 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:46.943 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.943 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:46.943 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.943 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:46.943 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:46.943 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.943 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:46.944 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:46.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:46.944 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:46.944 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:46.944 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:46.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:46.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:46.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:46.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.944 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:46.944 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:46.945 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:46.945 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:46.945 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.945 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:46.945 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:46.945 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:46.945 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.945 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:46.945 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:46.945 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:46.945 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.945 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:46.945 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.945 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:46.945 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.945 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:46.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:46.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:46.946 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:46.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:46.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:46.946 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:46.946 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:46.946 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:46.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:46.946 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.947 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:46.947 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:46.947 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:46.947 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:46.947 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:46.947 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.947 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:46.947 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.947 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:46.947 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:46.947 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:46.947 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:46.947 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:46.947 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:46.947 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_asio 2024-03-28T16:16:46.948 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:46.948 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:46.948 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40116 2024-03-28T16:16:46.948 INFO:tasks.workunit.client.0.smithi007.stdout:test api_asio on pid 40116 2024-03-28T16:16:46.955 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_asio on pid 40116' 2024-03-28T16:16:46.955 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-03-28T16:16:46.955 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40116 2024-03-28T16:16:46.955 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:46.955 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:46.955 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:46.955 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_list 2024-03-28T16:16:46.955 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_list' 2024-03-28T16:16:46.955 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_list 2024-03-28T16:16:46.956 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:46.956 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:46.956 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:46.956 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:46.956 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.956 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:46.956 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.956 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:46.956 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:46.956 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:46.956 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:46.956 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:46.956 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:46.956 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:46.957 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.957 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:46.957 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:46.957 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:46.957 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.957 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:46.957 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:46.957 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:46.957 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:46.957 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:46.957 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:46.957 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.957 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:46.958 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:46.958 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.958 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:46.958 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:46.958 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:46.958 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:46.958 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:46.958 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.958 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:46.958 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.958 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:46.958 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:46.958 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:46.958 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:46.959 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.959 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:46.959 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:46.959 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.959 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:46.959 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:46.959 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:46.959 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.959 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:46.959 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.959 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:46.959 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:46.959 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:46.959 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:46.959 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.960 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:46.960 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:46.960 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:46.960 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:46.960 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.960 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:46.960 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.960 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:46.960 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:46.960 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.960 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:46.960 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.960 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:46.960 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:46.960 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:46.960 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.961 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:46.961 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:46.961 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.961 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:46.961 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:46.961 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.961 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:46.961 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.961 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40074/exe 2024-03-28T16:16:46.961 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:46.961 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:46.961 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.961 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:46.961 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.961 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:46.962 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:46.962 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:46.962 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.962 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:46.962 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.962 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:46.962 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:46.962 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:46.962 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_list 2024-03-28T16:16:46.962 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40133 2024-03-28T16:16:46.962 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_list on pid 40133' 2024-03-28T16:16:46.962 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:46.963 INFO:tasks.workunit.client.0.smithi007.stdout:test api_list on pid 40133 2024-03-28T16:16:46.972 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40133 2024-03-28T16:16:46.972 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:46.972 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:46.973 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-03-28T16:16:46.973 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_lock 2024-03-28T16:16:46.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:46.973 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_lock' 2024-03-28T16:16:46.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:46.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:46.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:46.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:46.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:46.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:46.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:46.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:46.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:46.973 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:46.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.974 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:46.974 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:46.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:46.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:46.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:46.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:46.974 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:46.974 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:46.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:46.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.974 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:46.975 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:46.975 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:46.975 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:46.975 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.975 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:46.975 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.975 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:46.975 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:46.975 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:46.975 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:46.975 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_aio 2024-03-28T16:16:46.975 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:46.975 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:46.975 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:46.975 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.975 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:46.976 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_aio.log 2024-03-28T16:16:46.976 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_lock 2024-03-28T16:16:46.976 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_aio: /' 2024-03-28T16:16:46.976 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:46.976 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:46.976 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:46.976 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:46.976 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.976 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:46.976 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:46.976 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.976 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:46.976 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:46.976 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:46.976 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:46.977 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:46.977 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.977 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:46.977 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.977 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:46.977 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:46.977 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:46.977 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:46.977 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:46.977 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.977 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:46.977 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.977 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:46.977 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.977 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:46.977 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:46.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.978 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:46.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:46.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:46.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:46.978 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:46.978 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:46.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:46.978 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:46.978 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:46.978 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:46.978 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:46.979 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:46.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:46.979 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:46.979 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:46.979 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:46.979 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:46.979 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:46.979 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:46.979 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:46.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.979 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:46.979 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:46.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:46.979 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.980 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:46.980 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:46.980 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.980 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:46.980 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:46.980 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.980 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:46.980 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:46.980 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.980 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:46.980 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:46.980 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:46.980 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:46.980 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:46.980 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:46.981 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:46.981 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:46.981 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40081/exe 2024-03-28T16:16:46.981 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:46.981 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_lock 2024-03-28T16:16:46.981 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:46.981 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:46.981 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:46.981 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:46.981 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:46.981 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40156 2024-03-28T16:16:46.981 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:46.981 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:46.981 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:46.981 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-03-28T16:16:46.981 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:46.982 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_lock on pid 40156' 2024-03-28T16:16:46.982 INFO:tasks.workunit.client.0.smithi007.stdout:test api_lock on pid 40156 2024-03-28T16:16:46.990 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40156 2024-03-28T16:16:46.990 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:46.990 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:46.990 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:46.990 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:46.990 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:46.990 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:46.990 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.991 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:46.991 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:46.991 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.991 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:46.991 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:46.991 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:46.991 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:46.991 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.991 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:46.991 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:46.991 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:46.991 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.991 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:46.991 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.991 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:46.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:46.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:46.992 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:46.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:46.992 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:46.992 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:46.992 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:46.992 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:46.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:46.992 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.992 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:46.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:46.993 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:46.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:46.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:46.993 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:46.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:46.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:46.993 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_lock_pp 2024-03-28T16:16:46.993 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:46.993 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_lock_pp' 2024-03-28T16:16:46.993 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:46.994 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:46.994 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:46.994 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:46.994 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40090/exe 2024-03-28T16:16:46.994 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:46.994 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:46.994 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_lock_pp 2024-03-28T16:16:46.994 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:46.994 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:46.994 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:46.994 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:46.994 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.994 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:46.995 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:46.995 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.995 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:46.995 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:46.995 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.995 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-03-28T16:16:46.995 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:46.995 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:46.995 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:46.995 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.995 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:46.995 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.995 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:46.995 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:46.995 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:46.996 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:46.996 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.996 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:46.996 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.996 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:46.996 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:46.996 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:46.996 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:46.996 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:46.996 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:46.996 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:46.996 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:46.996 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:46.996 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:46.996 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.996 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:46.997 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:46.997 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_aio_pp 2024-03-28T16:16:46.997 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.997 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:46.997 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:46.997 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:46.997 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:46.997 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.997 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:46.997 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.997 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:46.997 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:46.997 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:46.997 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:46.997 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:46.997 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:46.998 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:46.998 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:46.998 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:46.998 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:46.998 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:46.998 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:46.998 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:46.998 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:46.998 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:46.998 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:46.998 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_lock_pp 2024-03-28T16:16:46.998 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40177 2024-03-28T16:16:46.998 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_lock_pp on pid 40177' 2024-03-28T16:16:46.999 INFO:tasks.workunit.client.0.smithi007.stdout:test api_lock_pp on pid 40177 2024-03-28T16:16:47.012 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.012 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.012 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.012 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40177 2024-03-28T16:16:47.012 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_io 2024-03-28T16:16:47.013 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.013 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.013 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.013 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.013 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.013 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.013 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:47.013 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:47.013 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.013 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.013 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.013 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.013 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.013 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.014 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.014 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.014 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.014 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.014 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.014 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.014 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-03-28T16:16:47.014 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_io.log 2024-03-28T16:16:47.014 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.014 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.014 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.014 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.014 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.014 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.015 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.015 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.015 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.015 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:47.015 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.015 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:47.015 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40101/exe 2024-03-28T16:16:47.015 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.015 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.015 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:47.015 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:47.015 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.015 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:47.015 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.015 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.016 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.016 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.016 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_aio_pp: /' 2024-03-28T16:16:47.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.016 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.016 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.016 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.016 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.017 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.017 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_io: /' 2024-03-28T16:16:47.017 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:47.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:47.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:47.017 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.017 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:47.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:47.017 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.017 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:47.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.017 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:47.018 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.018 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:47.018 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.018 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.018 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.018 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.018 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.018 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.018 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.018 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:47.018 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.018 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.018 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:47.018 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.018 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.019 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.019 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_misc 2024-03-28T16:16:47.019 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_misc' 2024-03-28T16:16:47.019 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_io_pp: /' 2024-03-28T16:16:47.019 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-03-28T16:16:47.019 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.019 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_io_pp 2024-03-28T16:16:47.019 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.019 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.019 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.019 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.019 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.019 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.019 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_misc 2024-03-28T16:16:47.019 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.020 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:47.020 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.020 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.020 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.020 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.020 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.020 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.021 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.021 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.021 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.021 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.021 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.021 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.021 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.022 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.022 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.022 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.022 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.022 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.022 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.022 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.022 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.022 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.022 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.022 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.022 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.022 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.022 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.022 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.023 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.023 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.023 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.023 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.023 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.023 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.023 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.023 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.023 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.023 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.023 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.023 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.023 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_misc 2024-03-28T16:16:47.023 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40202 2024-03-28T16:16:47.023 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_misc on pid 40202' 2024-03-28T16:16:47.024 INFO:tasks.workunit.client.0.smithi007.stdout:test api_misc on pid 40202 2024-03-28T16:16:47.032 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40202 2024-03-28T16:16:47.032 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:47.032 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:47.032 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-03-28T16:16:47.032 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.033 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.033 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.033 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.033 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.033 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.033 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.033 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.033 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.033 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.033 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.033 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.033 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.033 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.033 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.033 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.034 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.034 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.034 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.034 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:47.034 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:47.034 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.034 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.034 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.034 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:47.034 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:47.034 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.034 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:47.034 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.034 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_misc_pp 2024-03-28T16:16:47.034 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.035 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_misc_pp' 2024-03-28T16:16:47.035 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.035 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.035 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.035 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.035 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:47.035 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.035 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.035 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.035 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.035 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.035 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.035 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.035 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.035 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.036 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.036 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_misc_pp 2024-03-28T16:16:47.036 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.036 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.036 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.036 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.036 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.036 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.036 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:47.036 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:47.036 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.036 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.036 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.036 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.036 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.037 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.037 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:47.037 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.037 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40116/exe 2024-03-28T16:16:47.037 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.037 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:47.037 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:47.037 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.037 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:47.037 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.037 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:47.037 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.037 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.037 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.037 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:47.037 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.038 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:47.038 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.038 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.038 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:47.038 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.038 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.038 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.038 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.038 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.038 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.038 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.038 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.038 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.038 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.039 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.039 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.039 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.039 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.039 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.039 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.039 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.039 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_misc_pp 2024-03-28T16:16:47.039 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40220 2024-03-28T16:16:47.039 INFO:tasks.workunit.client.0.smithi007.stdout:test api_misc_pp on pid 40220 2024-03-28T16:16:47.047 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_misc_pp on pid 40220' 2024-03-28T16:16:47.047 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40220 2024-03-28T16:16:47.048 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:47.048 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:47.048 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_asio 2024-03-28T16:16:47.048 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_asio.log 2024-03-28T16:16:47.048 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.048 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.048 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.048 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.048 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.048 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.048 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.048 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.048 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.048 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.049 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.049 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.049 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.049 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.049 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.049 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.049 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.049 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.049 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.049 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.049 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.049 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.049 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.049 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.050 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.050 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.050 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.050 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.050 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-03-28T16:16:47.050 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.050 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.050 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.050 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.050 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.050 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.050 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.050 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.050 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.050 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.051 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.051 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.051 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.051 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.051 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40156/exe 2024-03-28T16:16:47.051 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:47.051 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:47.051 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:47.051 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:47.051 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.051 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.051 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:47.051 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_tier_pp 2024-03-28T16:16:47.051 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_tier_pp' 2024-03-28T16:16:47.051 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.052 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40133/exe 2024-03-28T16:16:47.052 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.052 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.052 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.052 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.052 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.052 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.052 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.052 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.052 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.052 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_tier_pp 2024-03-28T16:16:47.052 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_lock 2024-03-28T16:16:47.052 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.052 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_lock: /' 2024-03-28T16:16:47.052 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:47.053 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_lock.log 2024-03-28T16:16:47.053 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.053 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.053 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.053 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.053 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.053 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.053 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.053 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.053 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.053 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.053 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.053 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.053 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.053 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.053 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.054 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.054 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.054 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.054 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.054 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.054 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.054 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.054 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.054 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.054 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.054 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.054 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.054 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.054 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.055 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.055 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:47.055 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:47.055 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:47.055 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.055 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:47.055 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.055 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:47.055 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:47.055 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.055 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:47.055 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_asio: /' 2024-03-28T16:16:47.055 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.055 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:47.055 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.055 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.056 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.056 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:47.056 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.056 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:47.056 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.056 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_tier_pp 2024-03-28T16:16:47.056 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40241 2024-03-28T16:16:47.056 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_tier_pp on pid 40241' 2024-03-28T16:16:47.056 INFO:tasks.workunit.client.0.smithi007.stdout:test api_tier_pp on pid 40241 2024-03-28T16:16:47.065 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40241 2024-03-28T16:16:47.065 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:47.065 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:47.065 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-03-28T16:16:47.065 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.065 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.066 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.066 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.066 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.066 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_list: /' 2024-03-28T16:16:47.066 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.066 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.066 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.066 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.067 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.067 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.067 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.067 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.067 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.067 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.067 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.067 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.067 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.067 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.067 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.067 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.067 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.067 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.067 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:47.068 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:47.068 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:47.068 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:47.068 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.068 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:47.068 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.068 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.068 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.068 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.068 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.068 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.068 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.068 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.068 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.068 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.069 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.069 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.069 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.069 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.069 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.069 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.069 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.069 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.069 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.069 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.069 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.069 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.069 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.069 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.069 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.070 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.070 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.070 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.070 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.070 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.070 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.070 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.070 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.070 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.070 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.070 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.070 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_list.log 2024-03-28T16:16:47.070 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.070 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.070 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.071 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.071 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.071 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.071 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.071 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.071 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:47.071 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.071 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.071 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.071 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.071 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.071 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.071 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.071 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.071 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.072 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.072 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.072 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.072 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.072 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.072 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.072 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_pool 2024-03-28T16:16:47.072 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_pool' 2024-03-28T16:16:47.072 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_list 2024-03-28T16:16:47.072 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:47.072 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:47.072 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.072 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:47.072 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.072 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:47.072 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:47.073 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.073 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.073 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:47.073 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.073 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:47.073 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.073 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.073 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.073 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:47.073 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.073 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:47.073 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.073 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.073 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.073 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.074 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.074 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.074 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.074 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.074 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.074 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.074 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.075 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.075 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.075 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.075 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:19.552 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:47.076 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:02:19.617 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:47.076 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:02:19.622 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:47.078 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.078 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.078 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.078 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.078 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.078 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.078 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.078 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.078 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.079 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.079 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.079 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.079 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.079 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.079 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.079 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.079 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.079 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.079 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.079 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.079 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40202/exe 2024-03-28T16:16:47.079 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.079 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.079 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.080 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:19.556 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:47.085 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.085 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.085 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.085 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.085 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.085 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.086 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.086 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.086 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.086 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.086 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.086 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.086 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.086 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.086 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.086 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.086 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.086 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.086 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.086 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.087 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.087 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.087 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.087 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.087 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.087 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.087 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_pool 2024-03-28T16:16:47.087 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.087 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:47.087 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.087 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.087 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.087 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.087 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.087 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.087 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.088 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.088 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.088 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.088 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.088 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.088 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.088 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.088 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.088 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.088 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.088 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.088 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.088 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.088 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.088 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.089 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.089 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.089 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.089 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.089 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.089 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.089 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.089 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.089 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.089 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.089 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.089 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.089 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.089 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.089 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.089 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.090 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.090 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.090 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.090 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.090 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_pool 2024-03-28T16:16:47.090 INFO:tasks.workunit.client.0.smithi007.stdout:test api_pool on pid 40293 2024-03-28T16:16:47.092 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:02:19.647 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:47.104 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40293 2024-03-28T16:16:47.104 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_pool on pid 40293' 2024-03-28T16:16:47.104 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40293 2024-03-28T16:16:47.104 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:47.105 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:47.105 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-03-28T16:16:47.105 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:47.105 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:47.105 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.105 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:47.105 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:47.105 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.105 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:47.105 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.105 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.105 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.105 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.105 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.106 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.106 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.106 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.106 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40177/exe 2024-03-28T16:16:47.106 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.106 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.106 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.106 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.106 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.106 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.106 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.106 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.106 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.106 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.107 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.107 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:47.107 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.107 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:47.107 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:47.107 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.107 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:47.107 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.107 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:47.107 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:47.107 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.107 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:47.107 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.107 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:47.107 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.108 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.108 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.108 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:47.108 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.108 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:47.108 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.108 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_snapshots 2024-03-28T16:16:47.108 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_snapshots' 2024-03-28T16:16:47.108 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_misc: /' 2024-03-28T16:16:47.108 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40220/exe 2024-03-28T16:16:47.108 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_misc.log 2024-03-28T16:16:47.108 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.108 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_misc 2024-03-28T16:16:47.109 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.109 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.109 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.109 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.109 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.109 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.109 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.109 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.109 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.109 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.109 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-03-28T16:16:47.109 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.109 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_misc_pp: /' 2024-03-28T16:16:47.109 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.109 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.110 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_misc_pp 2024-03-28T16:16:47.110 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.110 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.110 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.110 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.110 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.110 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.110 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.110 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.110 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.110 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.110 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.110 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.110 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.110 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.111 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.111 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.111 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.111 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.111 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.111 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_lock_pp 2024-03-28T16:16:47.111 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.111 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_lock_pp: /' 2024-03-28T16:16:47.111 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.111 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.111 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.111 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.111 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.111 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.111 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.112 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.112 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.112 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.112 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.112 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.112 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.112 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.112 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.112 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.112 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.112 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.112 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.112 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.112 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.112 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.113 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.113 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.113 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:47.113 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.113 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_snapshots 2024-03-28T16:16:47.113 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.113 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.113 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.113 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.113 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.113 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.113 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.113 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.113 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.113 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.113 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.114 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.114 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.114 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.114 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.114 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.114 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.114 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.114 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.114 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.114 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.114 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.114 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.114 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.114 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.114 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.115 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.115 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.115 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.115 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.115 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.115 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.115 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.115 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.115 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.115 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.115 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-03-28T16:16:47.115 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_snapshots 2024-03-28T16:16:47.115 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40345 2024-03-28T16:16:47.115 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_snapshots on pid 40345' 2024-03-28T16:16:47.116 INFO:tasks.workunit.client.0.smithi007.stdout:test api_snapshots on pid 40345 2024-03-28T16:16:47.119 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40345 2024-03-28T16:16:47.120 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:47.120 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:47.120 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_snapshots_pp 2024-03-28T16:16:47.120 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.120 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.120 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.120 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.120 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.120 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.120 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.120 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.120 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.120 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_snapshots_pp' 2024-03-28T16:16:47.121 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.121 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.121 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.121 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40241/exe 2024-03-28T16:16:47.121 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.121 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.121 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.121 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.121 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.121 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.121 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:47.121 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.121 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.121 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.121 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.121 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.122 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.122 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.122 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.122 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.122 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.122 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.122 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_snapshots_pp 2024-03-28T16:16:47.122 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_tier_pp 2024-03-28T16:16:47.122 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_snapshots_pp 2024-03-28T16:16:47.122 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40359 2024-03-28T16:16:47.122 INFO:tasks.workunit.client.0.smithi007.stdout:test api_snapshots_pp on pid 40359 2024-03-28T16:16:47.132 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_snapshots_pp on pid 40359' 2024-03-28T16:16:47.133 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40359 2024-03-28T16:16:47.133 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:47.133 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:47.133 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-03-28T16:16:47.133 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-03-28T16:16:47.133 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.133 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.133 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.133 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.133 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.133 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.133 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:47.133 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:47.133 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:47.134 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:47.134 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.134 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:47.134 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:47.134 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:47.134 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:47.134 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.134 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:47.134 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.134 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:47.134 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:47.134 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.134 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:47.134 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.135 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:47.135 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.135 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.135 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.135 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:47.135 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.135 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:47.135 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.135 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_tier_pp: /' 2024-03-28T16:16:47.135 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-03-28T16:16:47.135 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.135 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.135 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.135 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.135 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.136 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.136 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.136 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.136 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.136 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.136 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.136 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.136 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.136 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.136 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.136 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.136 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.136 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.136 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.136 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.137 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.137 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:47.137 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:47.137 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:47.137 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:47.137 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.137 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:47.137 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_stat 2024-03-28T16:16:47.137 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_stat' 2024-03-28T16:16:47.137 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.137 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.137 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.137 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.137 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.137 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.137 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.138 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.138 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.138 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.138 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.138 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.138 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.138 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.138 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.138 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.138 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.138 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.138 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.138 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.138 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:47.138 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:47.139 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:47.139 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.139 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:47.139 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.139 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:47.139 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:47.139 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:47.139 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.139 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:47.139 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_stat 2024-03-28T16:16:47.139 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.139 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:47.139 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.139 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.139 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.140 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:47.140 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.140 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:47.140 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.140 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_stat 2024-03-28T16:16:47.140 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40396 2024-03-28T16:16:47.140 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_stat on pid 40396' 2024-03-28T16:16:47.140 INFO:tasks.workunit.client.0.smithi007.stdout:test api_stat on pid 40396 2024-03-28T16:16:47.149 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40396 2024-03-28T16:16:47.149 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:47.149 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:47.149 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40293/exe 2024-03-28T16:16:47.149 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-03-28T16:16:47.149 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.149 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.149 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.149 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.149 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.149 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.149 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.149 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.150 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.150 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.150 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.150 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.150 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.150 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.150 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.150 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.150 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.150 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.150 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_pool 2024-03-28T16:16:47.150 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.150 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.150 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_pool.log 2024-03-28T16:16:47.150 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_stat_pp 2024-03-28T16:16:47.151 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_pool: /' 2024-03-28T16:16:47.151 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.151 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.151 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.151 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.151 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.151 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.151 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.151 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.151 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.151 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.151 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.151 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.151 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.151 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.152 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.152 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.152 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.152 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.152 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.152 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.152 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.152 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.152 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.152 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.152 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.152 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.152 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:47.152 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_stat_pp' 2024-03-28T16:16:47.153 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:47.153 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.153 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.153 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.153 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:47.153 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:47.153 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.153 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:47.153 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.153 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.153 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.153 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.153 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:47.153 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.153 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.154 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.154 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_stat_pp 2024-03-28T16:16:47.154 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:47.154 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:47.154 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.154 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:47.154 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.154 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:47.154 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.154 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.154 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:47.154 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:47.154 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.154 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:47.154 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.154 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:47.155 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.155 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.155 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.155 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:47.155 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.155 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:47.155 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.155 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_stat_pp 2024-03-28T16:16:47.155 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40438 2024-03-28T16:16:47.155 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_stat_pp on pid 40438' 2024-03-28T16:16:47.155 INFO:tasks.workunit.client.0.smithi007.stdout:test api_stat_pp on pid 40438 2024-03-28T16:16:47.162 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40438 2024-03-28T16:16:47.162 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:47.163 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:47.163 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.163 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.163 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.163 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.163 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.163 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-03-28T16:16:47.163 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.163 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:47.163 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.163 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:47.163 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.163 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.163 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.164 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.164 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.164 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.164 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.164 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.164 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.164 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.164 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.164 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.164 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.164 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.164 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.164 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:47.164 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.164 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:47.164 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.165 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.165 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.165 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.165 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:47.165 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.165 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.165 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.165 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.165 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.165 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.165 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.165 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.165 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.165 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:47.165 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:47.166 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:47.166 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.166 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:47.166 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.166 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:47.166 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:47.166 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.166 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:47.166 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.166 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:47.166 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.166 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.166 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.166 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:47.166 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.167 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:47.167 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.167 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.167 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_watch_notify 2024-03-28T16:16:47.167 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.167 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.167 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_watch_notify' 2024-03-28T16:16:47.167 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.167 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.167 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.167 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.167 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.167 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.167 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.167 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.168 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.168 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.168 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.168 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.168 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_watch_notify 2024-03-28T16:16:47.168 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.168 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:47.168 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.168 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.168 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.168 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.168 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.168 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.168 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.168 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.168 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.169 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.169 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_watch_notify 2024-03-28T16:16:47.169 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40471 2024-03-28T16:16:47.169 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_watch_notify on pid 40471' 2024-03-28T16:16:47.169 INFO:tasks.workunit.client.0.smithi007.stdout:test api_watch_notify on pid 40471 2024-03-28T16:16:47.181 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40471 2024-03-28T16:16:47.181 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:47.181 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:47.181 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-03-28T16:16:47.181 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.181 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.181 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.181 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.181 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.181 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.181 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.181 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.182 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.182 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.182 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.182 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.182 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.182 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.182 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.182 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.182 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.182 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.182 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.182 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.182 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.182 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.182 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.183 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.183 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.183 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.183 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.183 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.183 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.183 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_watch_notify_pp 2024-03-28T16:16:47.183 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.183 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.183 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.183 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.183 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.183 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.183 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.183 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.184 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.184 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.184 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_watch_notify_pp' 2024-03-28T16:16:47.184 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.184 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.184 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.184 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:47.184 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:47.184 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:47.184 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:47.184 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.184 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:47.184 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40345/exe 2024-03-28T16:16:47.184 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.184 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.185 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.185 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.185 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.185 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.185 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.185 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.185 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.185 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.185 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.185 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.185 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.185 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.185 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.185 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.185 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_watch_notify_pp 2024-03-28T16:16:47.185 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.186 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:47.186 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.186 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.186 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.186 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.186 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.186 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.186 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.186 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:47.186 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:47.186 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.186 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.186 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:47.186 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.186 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:47.187 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:47.187 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.187 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:47.187 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.187 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:47.187 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.187 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.187 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.187 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.187 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.187 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.187 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.187 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.187 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.187 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.188 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:47.188 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.188 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:47.188 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.188 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.188 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.188 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.188 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-03-28T16:16:47.188 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.188 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.188 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.188 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.188 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.188 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.188 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.188 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.189 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.189 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.189 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.189 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.189 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.189 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.189 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.189 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.189 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.189 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.189 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.189 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.189 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:47.189 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.189 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_snapshots: /' 2024-03-28T16:16:47.190 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.190 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.190 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.190 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.190 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.190 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.190 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.190 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_snapshots 2024-03-28T16:16:47.190 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.190 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.190 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.190 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.190 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.190 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.190 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.191 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.191 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40359/exe 2024-03-28T16:16:47.191 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_watch_notify_pp 2024-03-28T16:16:47.191 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40499 2024-03-28T16:16:47.191 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_watch_notify_pp on pid 40499' 2024-03-28T16:16:47.191 INFO:tasks.workunit.client.0.smithi007.stdout:test api_watch_notify_pp on pid 40499 2024-03-28T16:16:47.198 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40499 2024-03-28T16:16:47.198 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:47.198 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:47.198 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.198 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.199 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.199 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.199 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.199 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.199 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.199 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.199 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.199 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.199 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.199 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.199 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.199 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.199 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.199 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.199 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_cmd 2024-03-28T16:16:47.200 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.200 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.200 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.200 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.200 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.200 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.200 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.200 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.200 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.200 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.200 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.200 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.200 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.200 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.200 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.201 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.201 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.201 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.201 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.201 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.201 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-03-28T16:16:47.201 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.201 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.201 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.201 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.201 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.201 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.201 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_cmd' 2024-03-28T16:16:47.201 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_cmd 2024-03-28T16:16:47.201 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:47.202 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:47.202 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.202 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.202 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:47.202 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:47.202 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.202 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:47.202 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.202 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.202 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.202 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.202 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.202 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:47.202 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.203 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.203 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.203 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.203 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.203 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.203 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.203 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.203 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.203 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.203 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.203 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.203 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.203 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.203 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:47.203 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.204 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.204 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.204 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40396/exe 2024-03-28T16:16:47.204 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.204 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.204 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.204 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-03-28T16:16:47.204 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.204 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-03-28T16:16:47.204 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.204 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_cmd 2024-03-28T16:16:47.204 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40513 2024-03-28T16:16:47.204 INFO:tasks.workunit.client.0.smithi007.stdout:test api_cmd on pid 40513 2024-03-28T16:16:47.216 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_cmd on pid 40513' 2024-03-28T16:16:47.216 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40513 2024-03-28T16:16:47.216 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:47.216 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:47.216 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-03-28T16:16:47.216 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_snapshots_pp 2024-03-28T16:16:47.216 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.216 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.216 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.216 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.216 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.216 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:47.216 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:47.217 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.217 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.217 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.217 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.217 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.217 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:47.217 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.217 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:47.217 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.217 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.217 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:47.217 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.217 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:47.217 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.217 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:47.218 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.218 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.218 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.218 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.218 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.218 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.218 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:47.218 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.218 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:47.218 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.218 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.218 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.218 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.218 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.218 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.219 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.219 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.219 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.219 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.219 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.219 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.219 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.219 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.219 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.219 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.219 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.219 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.219 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.219 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.219 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.220 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.220 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.220 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.220 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.220 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.220 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.220 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.220 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:47.220 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.220 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.220 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:47.220 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.220 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.220 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.220 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:47.220 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:47.221 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.221 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:47.221 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_stat 2024-03-28T16:16:47.221 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.221 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.221 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.221 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.221 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.221 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.221 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.221 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.221 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.221 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.221 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.221 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.222 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40438/exe 2024-03-28T16:16:47.222 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.222 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.222 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.222 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.222 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.222 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.222 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_stat.log 2024-03-28T16:16:47.222 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_cmd_pp 2024-03-28T16:16:47.222 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:47.222 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_cmd_pp' 2024-03-28T16:16:47.222 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:47.222 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:47.222 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.222 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:47.223 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_cmd_pp 2024-03-28T16:16:47.223 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.223 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:47.223 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_stat: /' 2024-03-28T16:16:47.223 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:47.223 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.223 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:47.223 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.223 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:47.223 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.223 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.223 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.223 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:47.223 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.223 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:47.224 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.224 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.224 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.224 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.224 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.224 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.224 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.224 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.224 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.224 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.224 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.224 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.224 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.224 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.224 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.225 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.225 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.225 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.225 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.225 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.225 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.225 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.225 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.225 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.225 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.225 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.225 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.225 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.225 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.225 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.226 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.226 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:47.226 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_stat_pp 2024-03-28T16:16:47.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-03-28T16:16:47.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [----------] Global test environment set-up. 2024-03-28T16:16:47.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [----------] 12 tests from AsioRados 2024-03-28T16:16:47.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.068+0000 7ffb7d63f780 10 monclient: get_monmap_and_config 2024-03-28T16:16:47.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.068+0000 7ffb7d63f780 10 monclient: build_initial_monmap 2024-03-28T16:16:47.226 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.068+0000 7ffb7d63f780 10 monclient: monmap: 2024-03-28T16:16:47.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: epoch 0 2024-03-28T16:16:47.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:16:47.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: last_changed 0.000000 2024-03-28T16:16:47.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: created 0.000000 2024-03-28T16:16:47.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: min_mon_release 0 (unknown) 2024-03-28T16:16:47.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: election_strategy: 1 2024-03-28T16:16:47.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:16:47.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:16:47.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:16:47.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.069+0000 7ffb7d63f780 1 Processor -- start 2024-03-28T16:16:47.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.069+0000 7ffb7d63f780 1 -- start start 2024-03-28T16:16:47.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.069+0000 7ffb7d63f780 10 monclient: init 2024-03-28T16:16:47.227 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.070+0000 7ffb7d63f780 10 monclient: _reopen_session rank -1 2024-03-28T16:16:47.228 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.070+0000 7ffb7d63f780 10 monclient: _add_conns ranks=[1,2,0] 2024-03-28T16:16:47.228 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.070+0000 7ffb7d63f780 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e93ad04390 0x55e93ad02160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:16:47.228 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.070+0000 7ffb7d63f780 10 monclient(hunting): picked mon.noname-b con 0x55e93ad04390 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:16:47.228 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.070+0000 7ffb7d63f780 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x55e93ad052c0 0x55e93ad07300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:16:47.228 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.070+0000 7ffb7d63f780 10 monclient(hunting): picked mon.noname-c con 0x55e93ad052c0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:16:47.228 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.070+0000 7ffb7d63f780 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93ad9ace0 0x55e93ad938a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:16:47.228 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.070+0000 7ffb7d63f780 10 monclient(hunting): picked mon.noname-a con 0x55e93ad9ace0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:16:47.228 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.070+0000 7ffb7d63f780 10 monclient(hunting): start opening mon connection 2024-03-28T16:16:47.228 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.070+0000 7ffb7d63f780 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e93ac17b50 con 0x55e93ad9ace0 2024-03-28T16:16:47.228 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.070+0000 7ffb7d63f780 10 monclient(hunting): start opening mon connection 2024-03-28T16:16:47.228 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.070+0000 7ffb7d63f780 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e93ac177f0 con 0x55e93ad04390 2024-03-28T16:16:47.228 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.070+0000 7ffb7d63f780 10 monclient(hunting): start opening mon connection 2024-03-28T16:16:47.228 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.070+0000 7ffb7d63f780 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e93abfa510 con 0x55e93ad052c0 2024-03-28T16:16:47.228 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.070+0000 7ffb7d63f780 10 monclient(hunting): _renew_subs 2024-03-28T16:16:47.228 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.070+0000 7ffb7d63f780 10 monclient(hunting): authenticate will time out at 1474.211345s 2024-03-28T16:16:47.231 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.231 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-03-28T16:16:47.231 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.231 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.231 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.231 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.231 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.231 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.231 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.231 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.231 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.231 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.231 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.232 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.232 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.232 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.232 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_stat_pp: /' 2024-03-28T16:16:47.232 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.232 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_cmd_pp 2024-03-28T16:16:47.232 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40564 2024-03-28T16:16:47.232 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_cmd_pp on pid 40564' 2024-03-28T16:16:47.232 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.100+test api_cmd_pp on pid 40564 2024-03-28T16:16:47.245 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40564 2024-03-28T16:16:47.245 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:47.245 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:47.245 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.246 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.246 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-03-28T16:16:47.246 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.246 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.246 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.246 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_service 2024-03-28T16:16:47.246 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.246 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.246 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.246 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.246 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.246 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.246 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.247 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.247 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.247 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.247 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.247 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.247 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.247 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.247 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.247 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.247 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.247 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.247 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.247 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_service' 2024-03-28T16:16:47.247 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.247 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.247 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.248 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.248 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.248 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.248 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.248 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.248 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.248 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.248 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.248 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.248 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.248 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.248 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.248 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.248 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.248 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.249 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.249 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.249 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.249 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.249 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.249 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.249 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.249 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.249 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.249 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:47.249 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:47.249 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.249 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.249 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:47.249 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.249 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.250 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.250 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.250 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.250 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:47.250 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.250 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.250 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:47.250 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.250 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.250 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.250 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.250 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.250 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40471/exe 2024-03-28T16:16:47.250 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_service 2024-03-28T16:16:47.251 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.251 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:47.251 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:47.251 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:47.251 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.251 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:47.251 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.251 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.251 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.251 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.251 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.251 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:47.251 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.251 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.251 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.252 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.252 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.252 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.252 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:47.252 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.252 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:47.252 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.252 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:47.252 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.252 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.252 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.252 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.252 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:47.252 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.252 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:47.252 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.253 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.253 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.253 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.253 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_watch_notify 2024-03-28T16:16:47.253 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.253 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.253 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.253 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.253 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.253 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.253 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.253 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.253 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.253 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.253 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.253 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.254 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.254 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.254 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.254 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.254 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.254 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.254 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_watch_notify: /' 2024-03-28T16:16:47.254 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-03-28T16:16:47.254 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.254 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.254 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.254 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.254 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.254 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.254 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.255 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.255 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.255 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.255 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.255 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.255 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.255 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.255 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.255 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.255 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.255 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.255 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.255 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.255 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:47.255 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.256 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.256 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.256 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.256 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.256 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.256 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.256 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.256 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.256 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.256 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.256 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.256 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.256 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.256 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.256 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.256 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.257 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.257 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:19.697 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:47.257 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_service 2024-03-28T16:16:47.257 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40594 2024-03-28T16:16:47.258 INFO:tasks.workunit.client.0.smithi007.stdout:test api_service on pid 40594 2024-03-28T16:16:47.258 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40513/exe 2024-03-28T16:16:47.258 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_service on pid 40594' 2024-03-28T16:16:47.258 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40594 2024-03-28T16:16:47.258 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:47.259 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:47.259 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:19.698 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:47.264 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.264 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.264 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.264 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.265 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.265 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.265 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.265 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.265 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.265 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.265 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.265 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.265 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.265 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.265 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.265 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.265 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.265 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.265 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.266 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.266 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.266 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.266 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.266 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.266 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.266 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.266 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.266 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_service_pp 2024-03-28T16:16:47.266 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.266 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.266 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.266 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_service_pp' 2024-03-28T16:16:47.266 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.267 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.267 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.267 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-03-28T16:16:47.267 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.267 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.267 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.267 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.267 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.267 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.267 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.267 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.267 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.267 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.267 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.267 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.268 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.268 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:47.268 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.268 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.268 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.268 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:47.268 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.268 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.268 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:47.268 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:47.268 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.268 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:47.268 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_cmd.log 2024-03-28T16:16:47.268 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:47.268 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_cmd 2024-03-28T16:16:47.269 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:47.269 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:47.269 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:47.269 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.269 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:47.269 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.269 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:47.269 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:19.702 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:47.271 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:47.271 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.271 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:47.271 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.271 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:47.271 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.271 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.271 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.272 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:47.272 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.272 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:47.272 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.272 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_service_pp 2024-03-28T16:16:47.272 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_service_pp 2024-03-28T16:16:47.272 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40610 2024-03-28T16:16:47.272 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_service_pp on pid 40610' 2024-03-28T16:16:47.272 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-03-28T16:16:47.272 INFO:tasks.workunit.client.0.smithi007.stdout:test api_service_pp on pid 40610 2024-03-28T16:16:47.276 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40610 2024-03-28T16:16:47.276 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:47.276 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:47.276 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_c_write_operations 2024-03-28T16:16:47.276 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_c_write_operations' 2024-03-28T16:16:47.276 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.276 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:47.276 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:47.276 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:47.277 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:47.277 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.277 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:47.277 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.277 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.277 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.277 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_c_write_operations 2024-03-28T16:16:47.277 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:47.277 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.277 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.277 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.277 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.277 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.277 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.277 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.278 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.278 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.278 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.278 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.278 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.278 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.278 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.278 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.278 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.278 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.278 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.278 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.278 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.278 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.278 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.279 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_cmd: /' 2024-03-28T16:16:47.279 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:47.279 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.279 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_c_write_operations 2024-03-28T16:16:47.279 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40628 2024-03-28T16:16:47.279 INFO:tasks.workunit.client.0.smithi007.stdout:test api_c_write_operations on pid 40628 2024-03-28T16:16:47.290 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_c_write_operations on pid 40628' 2024-03-28T16:16:47.290 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40628 2024-03-28T16:16:47.290 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:47.290 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:47.290 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-03-28T16:16:47.290 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:47.290 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:47.290 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.290 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:47.290 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.290 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.291 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.291 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.291 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.291 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.291 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.291 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.291 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.291 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.291 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:47.291 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.291 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:47.291 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.291 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.291 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.291 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:47.292 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.292 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.292 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.292 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.292 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.292 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.292 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.292 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.292 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.292 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.292 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.292 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.292 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.292 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.292 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.293 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.293 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:47.293 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_c_read_operations 2024-03-28T16:16:47.293 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.293 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.293 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.293 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:47.293 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.293 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:47.293 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.293 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.293 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.293 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:47.293 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:47.294 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_c_read_operations' 2024-03-28T16:16:47.294 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.294 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.294 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.294 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.294 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:47.294 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:47.294 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.294 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:47.294 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.294 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.294 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.294 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.294 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.294 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.295 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.295 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.295 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.295 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.295 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.295 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.295 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.295 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.295 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.295 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.295 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.295 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.295 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.295 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:47.295 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:47.296 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:47.296 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.296 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:47.296 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.296 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:47.296 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_c_read_operations 2024-03-28T16:16:47.296 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:47.296 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:47.296 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.296 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:47.296 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.296 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:47.296 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40564/exe 2024-03-28T16:16:47.296 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.296 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.297 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.297 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:47.297 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.297 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:47.297 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.297 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.297 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40499/exe 2024-03-28T16:16:47.297 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.297 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.297 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.297 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.297 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.297 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.297 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.297 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.298 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.298 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.298 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.298 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.298 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.298 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.298 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.298 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.298 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.298 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.298 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.298 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.298 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.298 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.298 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.299 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.299 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_c_read_operations 2024-03-28T16:16:47.299 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40656 2024-03-28T16:16:47.299 INFO:tasks.workunit.client.0.smithi007.stdout:test api_c_read_operations on pid 40656 2024-03-28T16:16:47.309 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_c_read_operations on pid 40656' 2024-03-28T16:16:47.309 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40656 2024-03-28T16:16:47.309 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:47.309 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:47.309 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.309 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.309 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.309 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-03-28T16:16:47.310 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_cmd_pp 2024-03-28T16:16:47.310 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.310 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s api_cls_remote_reads 2024-03-28T16:16:47.310 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' api_cls_remote_reads' 2024-03-28T16:16:47.310 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.310 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.310 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.310 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.310 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.310 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.310 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:47.310 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.310 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.310 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-03-28T16:16:47.310 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.311 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.311 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.311 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.311 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.311 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.311 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.311 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.311 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.311 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.311 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.311 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.311 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.311 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.311 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.311 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.311 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.312 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.312 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.312 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.312 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.312 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.312 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.312 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-03-28T16:16:47.312 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.312 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:47.312 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.312 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_watch_notify_pp 2024-03-28T16:16:47.312 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.312 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.312 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.312 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.312 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.313 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.313 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.313 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.313 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.313 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.313 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.313 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.313 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.313 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.313 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.313 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.313 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.313 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.313 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:47.313 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:47.313 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:47.314 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.314 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:47.314 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-03-28T16:16:47.314 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-03-28T16:16:47.314 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:47.314 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:47.314 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:47.314 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.314 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:47.314 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.314 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:47.314 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:47.314 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.314 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:47.314 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.315 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:47.315 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.315 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.315 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.315 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:47.315 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.315 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:47.315 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.315 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo api_cls_remote_reads 2024-03-28T16:16:47.315 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.315 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.315 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.315 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.315 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.315 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.315 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.316 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.316 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.316 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.316 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.316 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=api_cls_remote_reads 2024-03-28T16:16:47.316 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.316 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40686 2024-03-28T16:16:47.316 INFO:tasks.workunit.client.0.smithi007.stdout:test api_cls_remote_reads on pid 40686 2024-03-28T16:16:47.325 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test api_cls_remote_reads on pid 40686' 2024-03-28T16:16:47.325 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40686 2024-03-28T16:16:47.326 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:47.326 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:47.326 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.326 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-03-28T16:16:47.326 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.326 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.326 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.326 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.326 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.326 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.326 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.326 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.326 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.326 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.327 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.327 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:47.327 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:47.327 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:47.327 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:47.327 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.327 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:47.327 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.327 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.327 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.327 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.327 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.327 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:47.327 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.327 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.327 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.328 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.328 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s list_parallel 2024-03-28T16:16:47.328 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.328 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.328 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.328 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.328 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' list_parallel' 2024-03-28T16:16:47.328 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.328 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.328 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.328 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.328 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:47.328 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:47.328 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.328 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.329 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.329 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.329 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:47.329 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.329 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.329 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:47.329 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:47.329 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.329 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:47.329 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.329 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:47.329 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.329 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.329 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.329 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:47.330 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.330 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:47.330 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.330 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo list_parallel 2024-03-28T16:16:47.330 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.330 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.330 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.330 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.330 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:47.330 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.330 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.330 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.330 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.330 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.330 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.331 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.331 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.331 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.331 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.331 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.331 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.331 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.331 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.331 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.331 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.331 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.331 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.331 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.331 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.332 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.332 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.332 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.332 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.332 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.332 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.332 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.332 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.332 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.332 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.332 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.332 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.332 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.332 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.332 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=list_parallel 2024-03-28T16:16:47.332 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.333 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.333 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.333 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.333 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.333 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.333 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40705 2024-03-28T16:16:47.333 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40594/exe 2024-03-28T16:16:47.333 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test list_parallel on pid 40705' 2024-03-28T16:16:47.333 INFO:tasks.workunit.client.0.smithi007.stdout:test list_parallel on pid 40705 2024-03-28T16:16:47.349 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40705 2024-03-28T16:16:47.349 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:47.349 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:47.349 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.349 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.350 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.350 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.350 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.350 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.350 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.350 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.350 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.350 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.350 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.350 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.350 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-03-28T16:16:47.350 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.350 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.350 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.351 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.351 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.351 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.351 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.351 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.351 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.351 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.351 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.351 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.351 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s open_pools_parallel 2024-03-28T16:16:47.351 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.351 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' open_pools_parallel' 2024-03-28T16:16:47.351 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.351 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.351 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.351 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.352 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.352 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.352 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.352 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.352 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.352 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.352 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:47.352 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.352 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:47.352 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.352 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.352 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.352 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.352 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.352 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.352 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.353 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.353 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.353 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.353 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.353 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:47.353 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:47.353 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.353 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:47.353 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.353 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_service 2024-03-28T16:16:47.353 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.353 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.353 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.353 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.353 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.353 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.354 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.354 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.354 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.354 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.354 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.354 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.354 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.354 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.354 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.354 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.354 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.354 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.354 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.354 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.354 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.355 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.355 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.355 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.355 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.355 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.355 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.355 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.355 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40628/exe 2024-03-28T16:16:47.355 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_service.log 2024-03-28T16:16:47.355 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.355 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_service: /' 2024-03-28T16:16:47.355 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.355 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.355 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.355 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.355 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.356 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.356 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:47.356 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.356 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.356 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.356 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:47.356 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.356 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.356 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:47.356 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.356 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.356 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:47.356 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.356 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:47.356 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.357 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.357 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:47.357 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.357 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.357 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.357 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.357 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.357 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:47.357 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.357 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:47.357 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.357 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.357 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.357 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.357 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.357 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.358 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.358 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.358 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.358 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.358 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:47.358 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.358 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:47.358 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.358 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.358 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.358 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.358 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.358 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.358 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.358 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.359 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.359 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.359 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.359 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.359 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.359 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.359 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.359 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.359 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_c_write_operations 2024-03-28T16:16:47.359 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40610/exe 2024-03-28T16:16:47.359 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.359 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-03-28T16:16:47.359 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.359 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.359 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.360 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.360 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.360 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.360 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.360 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.360 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.360 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.360 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-03-28T16:16:47.360 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo open_pools_parallel 2024-03-28T16:16:47.360 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:47.360 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-03-28T16:16:47.360 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.360 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.360 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.361 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.361 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.361 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.361 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.361 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.361 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.361 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.361 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.361 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.361 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.361 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.361 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.361 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.361 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.361 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.361 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.362 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.362 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.362 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=open_pools_parallel 2024-03-28T16:16:47.362 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.362 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.362 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40757 2024-03-28T16:16:47.362 INFO:tasks.workunit.client.0.smithi007.stdout:test open_pools_parallel on pid 40757 2024-03-28T16:16:47.369 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test open_pools_parallel on pid 40757' 2024-03-28T16:16:47.369 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40757 2024-03-28T16:16:47.369 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-28T16:16:47.369 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:47.369 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-03-28T16:16:47.369 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.369 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.370 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.370 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.370 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.370 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.370 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_service_pp 2024-03-28T16:16:47.370 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.370 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.370 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.370 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.370 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.370 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.370 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.370 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.370 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.370 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:47.371 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:47.371 INFO:tasks.workunit.client.0.smithi007.stderr:++ printf %25s delete_pools_parallel 2024-03-28T16:16:47.371 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:47.371 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:47.371 INFO:tasks.workunit.client.0.smithi007.stderr:+ r=' delete_pools_parallel' 2024-03-28T16:16:47.371 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.371 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:47.371 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:47.371 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_service_pp: /' 2024-03-28T16:16:47.371 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:47.371 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:47.371 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.371 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.371 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.371 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.371 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.372 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:47.372 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.372 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:47.372 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.372 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.372 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:47.372 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.372 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:47.372 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.372 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:47.372 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.372 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.372 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.372 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.372 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.373 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.373 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:47.373 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.373 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:47.373 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.373 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.373 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.373 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.373 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.373 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.373 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.373 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.373 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.373 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.373 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.373 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.374 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.374 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.374 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.374 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.374 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.374 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.374 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.374 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.374 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.374 INFO:tasks.workunit.client.0.smithi007.stdout:0000 7ffb7b13e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e93ad04390 0x55e93ad02160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:16:47.374 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.111+0000 7ffb7b13e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e93ad04390 0x55e93ad02160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:42372/0 (socket says 172.21.15.7:42372) 2024-03-28T16:16:47.374 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.111+0000 7ffb7b13e640 1 -- 172.21.15.7:0/911623689 learned_addr learned my addr 172.21.15.7:0/911623689 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:16:47.375 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.111+0000 7ffb7b13e640 10 monclient(hunting): get_auth_request con 0x55e93ad04390 auth_method 0 2024-03-28T16:16:47.375 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.111+0000 7ffb7b13e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:16:47.375 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.111+0000 7ffb7b13e640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:16:47.375 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.111+0000 7ffb7b13e640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:16:47.375 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.115+0000 7ffb7b13e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:16:47.375 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.115+0000 7ffb7b13e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:16:47.375 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.115+0000 7ffb7b13e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:16:47.375 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.130+0000 7ffb7b13e640 10 monclient(hunting): handle_auth_done global_id 4444 payload 274 2024-03-28T16:16:47.375 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.130+0000 7ffb7b13e640 1 -- 172.21.15.7:0/911623689 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x55e93ad052c0 msgr2=0x55e93ad07300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:16:47.375 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.130+0000 7ffb7b13e640 1 --2- 172.21.15.7:0/911623689 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x55e93ad052c0 0x55e93ad07300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:16:47.375 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.130+0000 7ffb7b13e640 1 -- 172.21.15.7:0/911623689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93ad9ace0 msgr2=0x55e93ad938a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T16:16:47.375 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.130+0000 7ffb7b13e640 1 --2- 172.21.15.7:0/911623689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93ad9ace0 0x55e93ad938a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:16:47.375 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.130+0000 7ffb7b13e640 10 monclient: _finish_hunting 0 2024-03-28T16:16:47.375 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.130+0000 7ffb7b13e640 1 monclient: found mon.noname-b 2024-03-28T16:16:47.375 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.130+0000 7ffb7b13e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:16:47.376 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.130+0000 7ffb7b13e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:16:47.376 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.130+0000 7ffb7b13e640 1 -- 172.21.15.7:0/911623689 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6c005de0 con 0x55e93ad04390 2024-03-28T16:16:47.376 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.172+0000 7ffb7b13e640 1 --2- 172.21.15.7:0/911623689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e93ad04390 0x55e93ad02160 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c055040 tx=0x7ffb6c04c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81879bc5d5eb67ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:16:47.377 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.377 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.377 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.377 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.377 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.377 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.377 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.377 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.377 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.378 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.378 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.378 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.202+0000 7ffb7993b640 1 -- 172.21.15.7:0/911623689 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6c052640 con 0x55e93ad04390 2024-03-28T16:16:47.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.202+0000 7ffb7993b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:16:47.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.202+0000 7ffb7993b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:16:47.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.202+0000 7ffb7993b640 10 monclient: dump: 2024-03-28T16:16:47.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: epoch 1 2024-03-28T16:16:47.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:16:47.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:16:47.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:16:47.378 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: min_mon_release 18 (reef) 2024-03-28T16:16:47.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: election_strategy: 1 2024-03-28T16:16:47.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:16:47.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:16:47.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:16:47.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.202+0000 7ffb7993b640 10 monclient: _finish_auth 0 2024-03-28T16:16:47.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.202+0000 7ffb7993b640 10 monclient: _check_auth_tickets 2024-03-28T16:16:47.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.202+0000 7ffb7993b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:16:47.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.202+0000 7ffb7993b640 1 -- 172.21.15.7:0/911623689 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb6c0527a0 con 0x55e93ad04390 2024-03-28T16:16:47.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.202+0000 7ffb7993b640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:16:47.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.202+0000 7ffb7993b640 1 -- 172.21.15.7:0/911623689 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6c052b50 con 0x55e93ad04390 2024-03-28T16:16:47.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.202+0000 7ffb7993b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:16:47.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.202+0000 7ffb7993b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:16:47.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.202+0000 7ffb7993b640 10 monclient: dump: 2024-03-28T16:16:47.379 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: epoch 1 2024-03-28T16:16:47.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:16:47.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:16:47.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:16:47.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: min_mon_release 18 (reef) 2024-03-28T16:16:47.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: election_strategy: 1 2024-03-28T16:16:47.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:16:47.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:16:47.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:16:47.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.202+0000 7ffb7d63f780 5 monclient: authenticate success, global_id 4444 2024-03-28T16:16:47.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.202+0000 7ffb7d63f780 10 monclient: get_monmap_and_config success 2024-03-28T16:16:47.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.202+0000 7ffb7d63f780 4 set_mon_vals no callback set 2024-03-28T16:16:47.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.203+0000 7ffb7d63f780 10 monclient: shutdown 2024-03-28T16:16:47.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.203+0000 7ffb7d63f780 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:16:47.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.203+0000 7ffb7d63f780 1 -- 172.21.15.7:0/911623689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e93ad04390 msgr2=0x55e93ad02160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:16:47.380 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.203+0000 7ffb7d63f780 1 --2- 172.21.15.7:0/911623689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e93ad04390 0x55e93ad02160 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c055040 tx=0x7ffb6c04c000 comp rx=0 tx=0).stop 2024-03-28T16:16:47.382 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.382 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.382 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.382 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.382 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.382 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.382 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.382 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.382 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.382 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.382 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.382 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.383 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.383 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.383 INFO:tasks.workunit.client.0.smithi007.stderr:++ awk '{print $1}' 2024-03-28T16:16:47.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.203+0000 7ffb7d63f780 1 -- 172.21.15.7:0/911623689 shutdown_connections 2024-03-28T16:16:47.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.203+0000 7ffb7d63f780 1 --2- 172.21.15.7:0/911623689 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93ad9ace0 0x55e93ad938a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:16:47.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.203+0000 7ffb7d63f780 1 --2- 172.21.15.7:0/911623689 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x55e93ad052c0 0x55e93ad07300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:16:47.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.203+0000 7ffb7d63f780 1 --2- 172.21.15.7:0/911623689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e93ad04390 0x55e93ad02160 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:16:47.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.203+0000 7ffb7d63f780 1 -- 172.21.15.7:0/911623689 >> 172.21.15.7:0/911623689 conn(0x55e93ad08560 msgr2=0x55e93ad9bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:16:47.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.204+0000 7ffb7d63f780 1 -- 172.21.15.7:0/911623689 shutdown_connections 2024-03-28T16:16:47.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.204+0000 7ffb7d63f780 1 -- 172.21.15.7:0/911623689 wait complete. 2024-03-28T16:16:47.383 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.204+0000 7ffb7d63f780 10 monclient: build_initial_monmap 2024-03-28T16:16:47.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.204+0000 7ffb7d63f780 10 monclient: monmap: 2024-03-28T16:16:47.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: epoch 0 2024-03-28T16:16:47.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-03-28T16:16:47.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: last_changed 0.000000 2024-03-28T16:16:47.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: created 0.000000 2024-03-28T16:16:47.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: min_mon_release 0 (unknown) 2024-03-28T16:16:47.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: election_strategy: 1 2024-03-28T16:16:47.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2024-03-28T16:16:47.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-b 2024-03-28T16:16:47.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2024-03-28T16:16:47.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.204+0000 7ffb7d63f780 1 librados: starting msgr at 2024-03-28T16:16:47.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.204+0000 7ffb7d63f780 1 librados: starting objecter 2024-03-28T16:16:47.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.204+0000 7ffb7d63f780 1 Processor -- start 2024-03-28T16:16:47.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.205+0000 7ffb7d63f780 1 -- start start 2024-03-28T16:16:47.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.205+0000 7ffb7d63f780 1 librados: setting wanted keys 2024-03-28T16:16:47.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.205+0000 7ffb7d63f780 1 librados: calling monclient init 2024-03-28T16:16:47.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.205+0000 7ffb7d63f780 10 monclient: init 2024-03-28T16:16:47.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.205+0000 7ffb7d63f780 10 monclient: _reopen_session rank -1 2024-03-28T16:16:47.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.205+0000 7ffb7d63f780 10 monclient: _add_conns ranks=[2,0,1] 2024-03-28T16:16:47.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.205+0000 7ffb7d63f780 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x55e93ad04390 0x55e93ad914c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:16:47.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.205+0000 7ffb7d63f780 10 monclient(hunting): picked mon.noname-c con 0x55e93ad04390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T16:16:47.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.205+0000 7ffb7d63f780 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93ad052c0 0x55e93ad99e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:16:47.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.205+0000 7ffb7d63f780 10 monclient(hunting): picked mon.noname-a con 0x55e93ad052c0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2024-03-28T16:16:47.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.205+0000 7ffb7d63f780 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e93ad9ace0 0x55e93ad90ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:16:47.387 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.387 INFO:tasks.workunit.client.0.smithi007.stderr:++ echo delete_pools_parallel 2024-03-28T16:16:47.387 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.387 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.387 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.387 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.388 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.388 INFO:tasks.workunit.client.0.smithi007.stderr:+ ff=delete_pools_parallel 2024-03-28T16:16:47.388 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.388 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.388 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40797 2024-03-28T16:16:47.388 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.388 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.388 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.388 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.388 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.388 INFO:tasks.workunit.client.0.smithi007.stderr:+ echo 'test delete_pools_parallel on pid 40797' 2024-03-28T16:16:47.388 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.388 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.388 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.388 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.388 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.389 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.205+0000 7ftest delete_pools_parallel on pid 40797 2024-03-28T16:16:47.416 INFO:tasks.workunit.client.0.smithi007.stderr:+ pids[$f]=40797 2024-03-28T16:16:47.416 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.416 INFO:tasks.workunit.client.0.smithi007.stderr:+ ret=0 2024-03-28T16:16:47.416 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' 1 -eq 1 ']' 2024-03-28T16:16:47.416 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.416 INFO:tasks.workunit.client.0.smithi007.stderr:+ for t in "${!pids[@]}" 2024-03-28T16:16:47.416 INFO:tasks.workunit.client.0.smithi007.stderr:+ pid=40241 2024-03-28T16:16:47.417 INFO:tasks.workunit.client.0.smithi007.stderr:+ wait 40241 2024-03-28T16:16:47.417 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.417 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.417 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.417 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.417 INFO:tasks.workunit.client.0.smithi007.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-03-28T16:16:47.417 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.417 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.417 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.417 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.417 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.417 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.417 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.417 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.417 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.418 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.418 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.418 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.418 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.418 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.418 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -f /etc/bashrc ']' 2024-03-28T16:16:47.418 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.418 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.418 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.418 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.418 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.418 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.418 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.419 INFO:tasks.workunit.client.0.smithi007.stderr:+ . /etc/bashrc 2024-03-28T16:16:47.419 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.419 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.419 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -z '' ']' 2024-03-28T16:16:47.419 INFO:tasks.workunit.client.0.smithi007.stderr:++ BASHRCSOURCED=Y 2024-03-28T16:16:47.419 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.419 INFO:tasks.workunit.client.0.smithi007.stderr:++ shopt -q login_shell 2024-03-28T16:16:47.419 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.419 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.419 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.419 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.419 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.419 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.419 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.420 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.420 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.420 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.420 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.420 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.420 INFO:tasks.workunit.client.0.smithi007.stderr:+++ umask 2024-03-28T16:16:47.420 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.420 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.420 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.420 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.420 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.420 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.420 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' 0022 -eq 0 ']' 2024-03-28T16:16:47.420 INFO:tasks.workunit.client.0.smithi007.stderr:++ SHELL=/bin/bash 2024-03-28T16:16:47.420 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.421 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.421 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-28T16:16:47.421 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.421 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-28T16:16:47.421 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.421 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-28T16:16:47.421 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.421 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.421 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.421 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.421 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-28T16:16:47.421 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.421 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-28T16:16:47.421 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.421 INFO:tasks.workunit.client.0.smithi007.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-28T16:16:47.421 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.422 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-28T16:16:47.422 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.422 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-28T16:16:47.422 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.422 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.422 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.422 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.422 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.422 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.422 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.422 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.422 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.422 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.422 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.422 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.423 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.423 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.423 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.423 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.423 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.423 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.423 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.423 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40686/exe 2024-03-28T16:16:47.423 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.423 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.423 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.423 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.423 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.423 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.423 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.423 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.424 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.424 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.424 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.424 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.424 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.424 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40705/exe 2024-03-28T16:16:47.424 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40656/exe 2024-03-28T16:16:47.424 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.424 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.424 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.424 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.424 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.424 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.424 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.424 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.424 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.425 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.425 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.425 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.425 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.425 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.425 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.425 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.425 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.425 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.425 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.425 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.425 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.425 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.425 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.425 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.425 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.426 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.426 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.426 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.426 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.426 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.426 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.426 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.426 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.426 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.426 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'grep=grep --color=auto' 2024-03-28T16:16:47.426 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-28T16:16:47.426 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.426 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.426 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.427 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.427 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.427 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.427 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.427 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-28T16:16:47.427 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.427 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-03-28T16:16:47.427 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-28T16:16:47.427 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.427 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorls.sh 2024-03-28T16:16:47.427 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' '!' -t 0 ']' 2024-03-28T16:16:47.427 INFO:tasks.workunit.client.0.smithi007.stderr:+++ return 2024-03-28T16:16:47.427 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.427 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-28T16:16:47.427 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.427 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-28T16:16:47.428 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export S_COLORS=auto 2024-03-28T16:16:47.428 INFO:tasks.workunit.client.0.smithi007.stderr:+++ S_COLORS=auto 2024-03-28T16:16:47.428 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.428 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-28T16:16:47.428 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.428 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-28T16:16:47.428 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.428 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-03-28T16:16:47.428 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-03-28T16:16:47.428 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_c_read_operations 2024-03-28T16:16:47.428 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-03-28T16:16:47.428 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-28T16:16:47.428 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-28T16:16:47.428 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-28T16:16:47.428 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.429 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-28T16:16:47.429 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.429 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-28T16:16:47.429 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-28T16:16:47.429 INFO:tasks.workunit.client.0.smithi007.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-28T16:16:47.429 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_list_parallel 2024-03-28T16:16:47.429 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.429 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.429 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.429 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.429 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.429 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.429 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.429 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.429 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.429 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.430 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.430 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-28T16:16:47.430 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.430 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-28T16:16:47.430 INFO:tasks.workunit.client.0.smithi007.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-28T16:16:47.430 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.430 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.430 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.430 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.430 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.430 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.430 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-28T16:16:47.430 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.430 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-28T16:16:47.430 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-28T16:16:47.431 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.431 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-28T16:16:47.431 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.431 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/gawk.sh 2024-03-28T16:16:47.431 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_list_parallel.log 2024-03-28T16:16:47.431 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.431 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-28T16:16:47.431 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.431 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/lang.sh 2024-03-28T16:16:47.431 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup 2024-03-28T16:16:47.431 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ list_parallel: /' 2024-03-28T16:16:47.431 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.431 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/locale 2024-03-28T16:16:47.431 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.431 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.432 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.432 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.432 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.432 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.432 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.432 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-28T16:16:47.432 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.432 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-28T16:16:47.432 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-28T16:16:47.432 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.432 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.432 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.432 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.432 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.432 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.433 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.433 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.433 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_api_cls_remote_reads 2024-03-28T16:16:47.433 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.433 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.433 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.433 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.433 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.433 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.433 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.433 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.433 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.433 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.433 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.433 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.433 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.434 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40797/exe 2024-03-28T16:16:47.434 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.434 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.434 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.434 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.434 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.434 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.434 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.434 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.434 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.434 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.434 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.434 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_delete_pools_parallel 2024-03-28T16:16:47.434 INFO:tasks.workunit.client.0.smithi007.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-28T16:16:47.434 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-03-28T16:16:47.435 INFO:tasks.workunit.client.0.smithi007.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-28T16:16:47.435 INFO:tasks.workunit.client.0.smithi007.stderr:++++ export LANG=en_US.UTF-8 2024-03-28T16:16:47.435 INFO:tasks.workunit.client.0.smithi007.stderr:++++ LANG=en_US.UTF-8 2024-03-28T16:16:47.435 INFO:tasks.workunit.client.0.smithi007.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-28T16:16:47.435 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-28T16:16:47.435 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.435 INFO:tasks.workunit.client.0.smithi007.stderr:+++ LANG=en_US.UTF-8 2024-03-28T16:16:47.435 INFO:tasks.workunit.client.0.smithi007.stderr:+++ unset LANG_backup config 2024-03-28T16:16:47.435 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n '' ']' 2024-03-28T16:16:47.435 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-28T16:16:47.435 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' dumb = linux ']' 2024-03-28T16:16:47.435 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.435 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-28T16:16:47.435 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.435 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/less.sh 2024-03-28T16:16:47.436 INFO:tasks.workunit.client.0.smithi007.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-28T16:16:47.436 INFO:tasks.workunit.client.0.smithi007.stderr:++ for i in /etc/profile.d/*.sh 2024-03-28T16:16:47.436 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-28T16:16:47.436 INFO:tasks.workunit.client.0.smithi007.stderr:++ '[' '' ']' 2024-03-28T16:16:47.436 INFO:tasks.workunit.client.0.smithi007.stderr:++ . /etc/profile.d/which2.sh 2024-03-28T16:16:47.436 INFO:tasks.workunit.client.0.smithi007.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-28T16:16:47.436 INFO:tasks.workunit.client.0.smithi007.stderr:+++++ readlink /proc/40757/exe 2024-03-28T16:16:47.436 INFO:tasks.workunit.client.0.smithi007.stderr:++++ basename /usr/bin/bash 2024-03-28T16:16:47.436 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_declare='declare -f' 2024-03-28T16:16:47.436 INFO:tasks.workunit.client.0.smithi007.stderr:+++ which_opt=-f 2024-03-28T16:16:47.436 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export which_declare 2024-03-28T16:16:47.436 INFO:tasks.workunit.client.0.smithi007.stderr:+++ export -f which 2024-03-28T16:16:47.436 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset i 2024-03-28T16:16:47.436 INFO:tasks.workunit.client.0.smithi007.stderr:++ unset -f pathmunge 2024-03-28T16:16:47.436 INFO:tasks.workunit.client.0.smithi007.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-28T16:16:47.436 INFO:tasks.workunit.client.0.smithi007.stderr:+ export PATH 2024-03-28T16:16:47.437 INFO:tasks.workunit.client.0.smithi007.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-28T16:16:47.437 INFO:tasks.workunit.client.0.smithi007.stderr:+ unset rc 2024-03-28T16:16:47.437 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-03-28T16:16:47.437 INFO:tasks.workunit.client.0.smithi007.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-03-28T16:16:47.437 INFO:tasks.workunit.client.0.smithi007.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-03-28T16:16:47.437 INFO:tasks.workunit.client.0.smithi007.stderr:+ ceph_test_rados_open_pools_parallel 2024-03-28T16:16:47.437 INFO:tasks.workunit.client.0.smithi007.stdout:fb7d63f780 10 monclient(hunting): picked mon.noname-b con 0x55e93ad9ace0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-28T16:16:47.437 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.205+0000 7ffb7d63f780 10 monclient(hunting): start opening mon connection 2024-03-28T16:16:47.437 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.205+0000 7ffb7d63f780 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e93ad92af0 con 0x55e93ad052c0 2024-03-28T16:16:47.437 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.205+0000 7ffb7d63f780 10 monclient(hunting): start opening mon connection 2024-03-28T16:16:47.438 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.205+0000 7ffb7d63f780 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e93acc75b0 con 0x55e93ad9ace0 2024-03-28T16:16:47.438 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.205+0000 7ffb7d63f780 10 monclient(hunting): start opening mon connection 2024-03-28T16:16:47.438 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.205+0000 7ffb7d63f780 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e93ad91cf0 con 0x55e93ad04390 2024-03-28T16:16:47.438 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.205+0000 7ffb7d63f780 10 monclient(hunting): _renew_subs 2024-03-28T16:16:47.438 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.205+0000 7ffb7d63f780 10 monclient(hunting): authenticate will time out at 1474.346681s 2024-03-28T16:16:47.438 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.274+0000 7ffb7b93f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e93ad9ace0 0x55e93ad90ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:16:47.438 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.276+0000 7ffb7b93f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e93ad9ace0 0x55e93ad90ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.7:42464/0 (socket says 172.21.15.7:42464) 2024-03-28T16:16:47.438 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.276+0000 7ffb7b93f640 1 -- 172.21.15.7:0/3632289376 learned_addr learned my addr 172.21.15.7:0/3632289376 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-28T16:16:47.438 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.276+0000 7ffb7b93f640 10 monclient(hunting): get_auth_request con 0x55e93ad9ace0 auth_method 0 2024-03-28T16:16:47.438 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.276+0000 7ffb7b93f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-28T16:16:47.438 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.276+0000 7ffb7b93f640 10 monclient(hunting): _init_auth method 2 2024-03-28T16:16:47.438 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.276+0000 7ffb7b93f640 10 monclient(hunting): _init_auth creating new auth 2024-03-28T16:16:47.438 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.283+0000 7ffb7b93f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-28T16:16:47.438 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.283+0000 7ffb7b93f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-28T16:16:47.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.283+0000 7ffb7b93f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-28T16:16:47.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.286+0000 7ffb7b93f640 10 monclient(hunting): handle_auth_done global_id 4483 payload 824 2024-03-28T16:16:47.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.286+0000 7ffb7b93f640 1 -- 172.21.15.7:0/3632289376 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x55e93ad04390 msgr2=0x55e93ad914c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:16:47.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.286+0000 7ffb7b93f640 1 --2- 172.21.15.7:0/3632289376 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x55e93ad04390 0x55e93ad914c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:16:47.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.286+0000 7ffb7b93f640 1 -- 172.21.15.7:0/3632289376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93ad052c0 msgr2=0x55e93ad99e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:16:47.439 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:19.872 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:16:47.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.286+0000 7ffb7b93f640 1 --2- 172.21.15.7:0/3632289376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e93ad052c0 0x55e93ad99e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:16:47.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.286+0000 7ffb7b93f640 10 monclient: _finish_hunting 0 2024-03-28T16:16:47.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.286+0000 7ffb7b93f640 1 monclient: found mon.noname-b 2024-03-28T16:16:47.439 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.286+0000 7ffb7b93f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:16:47.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.286+0000 7ffb7b93f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3300/0 2024-03-28T16:16:47.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.286+0000 7ffb7b93f640 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e93ac40db0 con 0x55e93ad9ace0 2024-03-28T16:16:47.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.314+0000 7ffb7b93f640 1 --2- 172.21.15.7:0/3632289376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e93ad9ace0 0x55e93ad90ad0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ffb7000aa90 tx=0x7ffb7000af50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:16:47.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.314+0000 7ffb78939640 10 client.?.objecter ms_handle_connect 0x55e93ad9ace0 2024-03-28T16:16:47.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.314+0000 7ffb78939640 10 client.?.objecter resend_mon_ops 2024-03-28T16:16:47.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.358+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== mon.1 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb70014470 con 0x55e93ad9ace0 2024-03-28T16:16:47.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.358+0000 7ffb78939640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:16:47.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.358+0000 7ffb78939640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-28T16:16:47.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.358+0000 7ffb78939640 10 monclient: dump: 2024-03-28T16:16:47.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: epoch 1 2024-03-28T16:16:47.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:16:47.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:16:47.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:16:47.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: min_mon_release 18 (reef) 2024-03-28T16:16:47.440 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: election_strategy: 1 2024-03-28T16:16:47.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:16:47.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:16:47.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:16:47.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.359+0000 7ffb78939640 10 monclient: _finish_auth 0 2024-03-28T16:16:47.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.359+0000 7ffb78939640 10 monclient: _check_auth_tickets 2024-03-28T16:16:47.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.359+0000 7ffb78939640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:16:47.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.359+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== mon.1 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb700145d0 con 0x55e93ad9ace0 2024-03-28T16:16:47.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.359+0000 7ffb78939640 10 monclient: handle_config config(0 keys) v1 2024-03-28T16:16:47.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.359+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== mon.1 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb70014980 con 0x55e93ad9ace0 2024-03-28T16:16:47.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.359+0000 7ffb78939640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-28T16:16:47.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.359+0000 7ffb78939640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-28T16:16:47.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.359+0000 7ffb78939640 10 monclient: dump: 2024-03-28T16:16:47.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: epoch 1 2024-03-28T16:16:47.441 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:16:47.442 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: last_changed 2024-03-28T16:13:56.420665+0000 2024-03-28T16:16:47.442 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: created 2024-03-28T16:13:56.420665+0000 2024-03-28T16:16:47.592 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: min_mon_release 18 (reef) 2024-03-28T16:16:47.593 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: election_strategy: 1 2024-03-28T16:16:47.593 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-28T16:16:47.593 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 1: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-28T16:16:47.593 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-03-28T16:16:47.593 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.593 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.359+0000 7ffb7d63f780 5 monclient: authenticate success, global_id 4483 2024-03-28T16:16:47.593 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.359+0000 7ffb7d63f780 10 monclient: _renew_subs 2024-03-28T16:16:47.593 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.359+0000 7ffb7d63f780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:16:47.593 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.359+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e93ac26c10 con 0x55e93ad9ace0 2024-03-28T16:16:47.593 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.359+0000 7ffb7913a640 4 set_mon_vals no callback set 2024-03-28T16:16:47.593 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.359+0000 7ffb7d63f780 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:16:47.593 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.359+0000 7ffb7d63f780 10 monclient: _renew_subs 2024-03-28T16:16:47.593 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.359+0000 7ffb7d63f780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:16:47.593 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.359+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e93ac1a050 con 0x55e93ad9ace0 2024-03-28T16:16:47.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.359+0000 7ffb7d63f780 1 librados: init done 2024-03-28T16:16:47.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.359+0000 7ffb7d63f780 10 librados: wait_for_osdmap waiting 2024-03-28T16:16:47.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.587+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== mon.1 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 7) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ffb70011070 con 0x55e93ad9ace0 2024-03-28T16:16:47.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.588+0000 7ffb78939640 1 --2- 172.21.15.7:0/3632289376 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7ffb6806a7b0 0x7ffb6806cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:16:47.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.588+0000 7ffb7b13e640 1 --2- 172.21.15.7:0/3632289376 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7ffb6806a7b0 0x7ffb6806cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:16:47.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.588+0000 7ffb7b13e640 10 monclient: get_auth_request con 0x7ffb6806a7b0 auth_method 0 2024-03-28T16:16:47.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.589+0000 7ffb7b13e640 1 --2- 172.21.15.7:0/3632289376 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7ffb6806a7b0 0x7ffb6806cc70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c061000 tx=0x7ffb6c005ac0 comp rx=0 tx=0).ready entity=mgr.4162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:16:47.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.589+0000 7ffb78939640 10 client.4483.objecter ms_handle_connect 0x7ffb6806a7b0 2024-03-28T16:16:47.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.592+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== mon.1 v2:172.21.15.83:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ffb70056ed0 con 0x55e93ad9ace0 2024-03-28T16:16:47.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.592+0000 7ffb78939640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_map(28..28 src has 1..28) v4 2024-03-28T16:16:47.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.592+0000 7ffb78939640 3 client.4483.objecter handle_osd_map got epochs [28,28] > 0 2024-03-28T16:16:47.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.592+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding full epoch 28 2024-03-28T16:16:47.594 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.592+0000 7ffb78939640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-28T16:16:50.034 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.592+0000 7ffb7d63f780 10 librados: wait_for_osdmap done waiting 2024-03-28T16:16:50.034 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.592+0000 7ffb7d63f780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:16:50.034 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.592+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ffb6c04ac00 con 0x55e93ad9ace0 2024-03-28T16:16:50.034 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.719+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== mon.1 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb70052b10 con 0x55e93ad9ace0 2024-03-28T16:16:50.034 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.719+0000 7ffb78939640 10 monclient: handle_get_version_reply finishing 1 version 28 2024-03-28T16:16:50.034 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.719+0000 7ffb7993b640 10 client.4483.objecter _get_latest_version latest 28, have it 2024-03-28T16:16:50.034 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.719+0000 7ffb7d63f780 10 client.4483.objecter create_pool name=ceph_test_rados_api_asio 2024-03-28T16:16:50.035 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.719+0000 7ffb7d63f780 10 client.4483.objecter pool_op_submit 1 2024-03-28T16:16:50.035 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.719+0000 7ffb7d63f780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:16:50.035 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:47.719+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x55e93ac85f00 con 0x55e93ad9ace0 2024-03-28T16:16:50.035 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.016+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== mon.1 v2:172.21.15.83:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v30) v1 ==== 43+0+0 (secure 0 0 0) 0x7ffb70019020 con 0x55e93ad9ace0 2024-03-28T16:16:50.035 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.016+0000 7ffb78939640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 pool_op_reply(tid 1 (0) Success v30) v1 2024-03-28T16:16:50.035 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.016+0000 7ffb78939640 10 client.4483.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v30) v1 2024-03-28T16:16:50.035 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.016+0000 7ffb78939640 10 client.4483.objecter have request 1 at 0x55e93ad955a0 Op: create 2024-03-28T16:16:50.035 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.016+0000 7ffb78939640 20 client.4483.objecter waiting for client to reach epoch 30 before calling back 2024-03-28T16:16:50.035 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.016+0000 7ffb78939640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:16:50.035 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.016+0000 7ffb78939640 10 monclient: _renew_subs 2024-03-28T16:16:50.035 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.016+0000 7ffb78939640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:16:50.035 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.016+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=29}) v3 -- 0x7ffb680704f0 con 0x55e93ad9ace0 2024-03-28T16:16:50.035 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.016+0000 7ffb78939640 10 client.4483.objecter done 2024-03-28T16:16:50.035 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.032+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== mon.1 v2:172.21.15.83:3300/0 8 ==== osd_map(29..30 src has 1..30) v4 ==== 11503+0+0 (secure 0 0 0) 0x7ffb7006e1b0 con 0x55e93ad9ace0 2024-03-28T16:16:50.036 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.033+0000 7ffb78939640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_map(29..30 src has 1..30) v4 2024-03-28T16:16:50.036 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.033+0000 7ffb78939640 3 client.4483.objecter handle_osd_map got epochs [29,30] > 28 2024-03-28T16:16:50.036 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.033+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 29 2024-03-28T16:16:50.036 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.033+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 30 2024-03-28T16:16:50.036 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.033+0000 7ffb78939640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-28T16:16:52.206 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.033+0000 7ffb7d63f780 10 librados: set snap read head -> 1 2024-03-28T16:16:52.206 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.033+0000 7ffb7d63f780 10 librados: writefull oid=exist nspace= 2024-03-28T16:16:52.206 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.033+0000 7ffb7d63f780 10 client.4483.objecter _op_submit op 0x55e93ad92210 2024-03-28T16:16:52.206 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.033+0000 7ffb7d63f780 20 client.4483.objecter _calc_target epoch 30 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:16:52.206 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.033+0000 7ffb7d63f780 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:16:52.206 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.033+0000 7ffb7d63f780 10 client.4483.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [6,1] primary 6 2024-03-28T16:16:52.206 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.033+0000 7ffb7d63f780 1 --2- 172.21.15.7:0/3632289376 >> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] conn(0x55e93acc8180 0x55e93ad82b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:16:52.206 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.033+0000 7ffb7d63f780 20 client.4483.objecter _get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:16:52.206 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.033+0000 7ffb7d63f780 10 client.4483.objecter _op_submit oid exist '@19' '@19' [writefull 0~5 in=5b] tid 2 osd.6 2024-03-28T16:16:52.206 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.033+0000 7ffb7d63f780 20 client.4483.objecter get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:16:52.207 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.033+0000 7ffb7d63f780 15 client.4483.objecter _session_op_assign 6 2 2024-03-28T16:16:52.207 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.033+0000 7ffb7d63f780 15 client.4483.objecter _send_op 2 to 19.1 on osd.6 2024-03-28T16:16:52.207 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.033+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- osd_op(unknown.0.0:2 19.1 19:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) v8 -- 0x55e93ad89810 con 0x55e93acc8180 2024-03-28T16:16:52.207 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.033+0000 7ffb7d63f780 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:16:52.207 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.033+0000 7ffb7d63f780 5 client.4483.objecter 1 in flight 2024-03-28T16:16:52.207 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.185+0000 7ffb7a93d640 1 --2- 172.21.15.7:0/3632289376 >> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] conn(0x55e93acc8180 0x55e93ad82b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:16:52.207 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.185+0000 7ffb7a93d640 10 monclient: get_auth_request con 0x55e93acc8180 auth_method 0 2024-03-28T16:16:52.207 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.204+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:16:52.207 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.204+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:16:52.207 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.204+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:16:52.207 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.204+0000 7ffb637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:16:52.207 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.206+0000 7ffb78939640 10 client.4483.objecter ms_handle_connect 0x55e93acc8180 2024-03-28T16:16:52.207 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:50.206+0000 7ffb7a93d640 1 --2- 172.21.15.7:0/3632289376 >> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] conn(0x55e93acc8180 0x55e93ad82b20 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:16:52.207 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:51.204+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:16:52.207 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:51.204+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:16:52.207 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:51.204+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:16:52.208 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:52.205+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:16:59.145 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-03-28T16:16:59.145 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [----------] Global test environment set-up. 2024-03-28T16:16:59.145 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [----------] 4 tests from LibRadosService 2024-03-28T16:16:59.145 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-03-28T16:16:59.145 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5437 ms) 2024-03-28T16:16:59.145 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-03-28T16:16:59.145 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ OK ] LibRadosService.RegisterLate (198 ms) 2024-03-28T16:16:59.145 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-03-28T16:16:59.145 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: cluster: 2024-03-28T16:16:59.145 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: id: 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:16:59.145 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: health: HEALTH_WARN 2024-03-28T16:16:59.146 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 9 pool(s) do not have an application enabled 2024-03-28T16:16:59.146 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-03-28T16:16:59.146 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: services: 2024-03-28T16:16:59.146 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-03-28T16:16:59.146 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: mgr: x(active, since 117s), standbys: y 2024-03-28T16:16:59.146 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: osd: 8 osds: 8 up (since 74s), 8 in (since 2m) 2024-03-28T16:16:59.146 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-03-28T16:16:59.146 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: data: 2024-03-28T16:16:59.146 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: pools: 28 pools, 817 pgs 2024-03-28T16:16:59.146 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: objects: 2 objects, 449 KiB 2024-03-28T16:16:59.146 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-03-28T16:16:59.146 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: pgs: 82.130% pgs unknown 2024-03-28T16:16:59.147 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 16.034% pgs not active 2024-03-28T16:16:59.147 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 671 unknown 2024-03-28T16:16:59.147 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 118 creating+peering 2024-03-28T16:16:59.147 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 15 active+clean 2024-03-28T16:16:59.147 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 13 creating+activating 2024-03-28T16:16:59.147 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-03-28T16:16:59.147 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-03-28T16:16:59.147 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: cluster: 2024-03-28T16:16:59.147 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: id: 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:16:59.147 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: health: HEALTH_WARN 2024-03-28T16:16:59.147 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 9 pool(s) do not have an application enabled 2024-03-28T16:16:59.147 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-03-28T16:16:59.147 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: services: 2024-03-28T16:16:59.147 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-03-28T16:16:59.147 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: mgr: x(active, since 119s), standbys: y 2024-03-28T16:16:59.148 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: osd: 8 osds: 8 up (since 76s), 8 in (since 2m) 2024-03-28T16:16:59.148 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-03-28T16:16:59.148 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: data: 2024-03-28T16:16:59.148 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: pools: 28 pools, 817 pgs 2024-03-28T16:16:59.148 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: objects: 2 objects, 449 KiB 2024-03-28T16:16:59.148 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-03-28T16:16:59.148 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: pgs: 82.130% pgs unknown 2024-03-28T16:16:59.148 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 16.034% pgs not active 2024-03-28T16:16:59.148 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 671 unknown 2024-03-28T16:16:59.148 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 118 creating+peering 2024-03-28T16:16:59.148 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 15 active+clean 2024-03-28T16:16:59.148 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 13 creating+activating 2024-03-28T16:16:59.148 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-03-28T16:16:59.148 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-03-28T16:16:59.148 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: cluster: 2024-03-28T16:16:59.149 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: id: 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T16:16:59.149 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: health: HEALTH_WARN 2024-03-28T16:16:59.149 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 8 pool(s) do not have an application enabled 2024-03-28T16:16:59.149 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-03-28T16:16:59.149 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: services: 2024-03-28T16:16:59.149 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-03-28T16:16:59.149 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-03-28T16:16:59.149 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: osd: 8 osds: 8 up (since 78s), 8 in (since 2m) 2024-03-28T16:16:59.149 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-03-28T16:16:59.149 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-03-28T16:16:59.149 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-03-28T16:16:59.149 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: data: 2024-03-28T16:16:59.149 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: pools: 30 pools, 881 pgs 2024-03-28T16:16:59.149 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: objects: 3 objects, 449 KiB 2024-03-28T16:16:59.149 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: usage: 215 MiB used, 720 GiB / 720 GiB avail 2024-03-28T16:16:59.150 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: pgs: 30.647% pgs unknown 2024-03-28T16:16:59.150 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 63.678% pgs not active 2024-03-28T16:17:02.363 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: api_asio: 2024-03-28T16:16:52.205+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:02.363 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:52.205+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:02.363 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:52.360+0000 7ffb7d3a2640 10 client.4483.objecter tick 2024-03-28T16:17:02.363 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:53.205+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:02.363 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:53.205+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:02.363 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:53.205+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:02.363 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:54.205+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:02.363 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:54.205+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:02.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:54.205+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:02.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:55.205+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:02.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:55.205+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:02.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:55.205+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:02.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:56.205+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:02.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:56.205+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:02.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:56.205+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:02.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:57.206+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:02.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:57.206+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:02.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:57.206+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:02.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:57.361+0000 7ffb7d3a2640 10 client.4483.objecter tick 2024-03-28T16:17:02.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:58.206+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:02.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:58.206+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:02.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:58.206+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:02.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:59.206+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:02.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:59.206+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:02.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:16:59.206+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:02.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:00.206+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:02.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:00.206+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:02.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:00.206+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:02.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:00.206+0000 7ffb637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:17:02.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:01.206+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:02.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:01.206+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:02.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:01.206+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:02.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.206+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:02.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.206+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:02.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.206+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:02.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.362+0000 7ffb7d3a2640 10 client.4483.objecter tick 2024-03-28T16:17:02.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.362+0000 7ffb7d3a2640 2 client.4483.objecter tid 2 on osd.6 is laggy 2024-03-28T16:17:02.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.362+0000 7ffb7d3a2640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:17:02.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.362+0000 7ffb7d3a2640 10 monclient: _renew_subs 2024-03-28T16:17:02.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.362+0000 7ffb7d3a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:17:02.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.362+0000 7ffb7d3a2640 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=31}) v3 -- 0x7ffb4c002f90 con 0x55e93ad9ace0 2024-03-28T16:17:02.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.362+0000 7ffb7d3a2640 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- ping magic: 0 v1 -- 0x7ffb4c003320 con 0x55e93acc8180 2024-03-28T16:17:02.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== mon.1 v2:172.21.15.83:3300/0 9 ==== osd_map(31..39 src has 1..39) v4 ==== 14393+0+0 (secure 0 0 0) 0x7ffb7005a600 con 0x55e93ad9ace0 2024-03-28T16:17:02.369 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_map(31..39 src has 1..39) v4 2024-03-28T16:17:02.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 3 client.4483.objecter handle_osd_map got epochs [31,39] > 30 2024-03-28T16:17:02.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 31 2024-03-28T16:17:02.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:02.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 31 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:02.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:02.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 32 2024-03-28T16:17:02.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:02.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 32 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:02.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:02.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 33 2024-03-28T16:17:02.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:02.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 33 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:02.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:02.371 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 34 2024-03-28T16:17:02.371 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:02.371 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 34 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:02.371 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:02.371 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 35 2024-03-28T16:17:02.371 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:02.371 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 35 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:02.371 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:07.340 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:39.913 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:07.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 36 2024-03-28T16:17:07.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:07.364 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 36 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:07.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:07.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 37 2024-03-28T16:17:07.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:07.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 37 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:07.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:07.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 38 2024-03-28T16:17:07.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:07.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 38 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:07.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:07.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 39 2024-03-28T16:17:07.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:07.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 39 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:07.367 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:07.367 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-28T16:17:07.367 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:02.365+0000 7ffb78939640 20 client.4483.objecter 2 19.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2024-03-28T16:17:07.367 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:03.206+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:07.367 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:03.206+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:07.367 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:03.206+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:07.368 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:04.206+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:07.368 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:04.206+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:07.368 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:04.206+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:07.368 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:05.206+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:07.368 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:05.206+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:07.368 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:05.206+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:07.368 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:06.206+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:07.368 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:06.206+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:07.369 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:06.206+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:07.369 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.206+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:07.369 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.206+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:07.369 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.206+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:07.369 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.363+0000 7ffb7d3a2640 10 client.4483.objecter tick 2024-03-28T16:17:07.399 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:39.972 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:07.422 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:02:39.979 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:07.427 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:02:39.983 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:07.489 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:40.062 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:07.542 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:40.115 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:07.622 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:02:40.178 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:07.657 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:40.230 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:07.825 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:40.398 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:08.208 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.363+0000 7ffb7d3a2640 2 client.4483.objecter tid 2 on osd.6 is laggy 2024-03-28T16:17:08.208 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.363+0000 7ffb7d3a2640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:17:08.208 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.363+0000 7ffb7d3a2640 10 monclient: _renew_subs 2024-03-28T16:17:08.208 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.363+0000 7ffb7d3a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:17:08.208 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.363+0000 7ffb7d3a2640 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=40}) v3 -- 0x7ffb4c0034e0 con 0x55e93ad9ace0 2024-03-28T16:17:08.208 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.363+0000 7ffb7d3a2640 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- ping magic: 0 v1 -- 0x7ffb4c003730 con 0x55e93acc8180 2024-03-28T16:17:08.208 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.425+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== mon.1 v2:172.21.15.83:3300/0 10 ==== osd_map(40..43 src has 1..43) v4 ==== 2546+0+0 (secure 0 0 0) 0x7ffb70008790 con 0x55e93ad9ace0 2024-03-28T16:17:08.208 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.425+0000 7ffb78939640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_map(40..43 src has 1..43) v4 2024-03-28T16:17:08.208 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.425+0000 7ffb78939640 3 client.4483.objecter handle_osd_map got epochs [40,43] > 39 2024-03-28T16:17:08.208 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.425+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 40 2024-03-28T16:17:08.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.425+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:08.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.425+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 40 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:08.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.425+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:08.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.425+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 41 2024-03-28T16:17:08.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.425+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:08.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.425+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 41 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:08.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.425+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:08.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.425+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 42 2024-03-28T16:17:08.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.425+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:08.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.425+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 42 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:08.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.425+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:08.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.425+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 43 2024-03-28T16:17:08.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.425+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:08.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.425+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 43 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:08.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.425+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:08.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.425+0000 7ffb78939640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-28T16:17:08.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:07.425+0000 7ffb78939640 20 client.4483.objecter 2 19.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2024-03-28T16:17:08.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:08.207+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:08.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:08.207+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:12.369 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:08.207+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:12.369 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:09.207+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:12.369 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:09.207+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:12.369 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:09.207+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:12.369 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:10.207+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:12.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:10.207+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:12.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:10.207+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:12.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:10.207+0000 7ffb637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:17:12.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:11.207+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:12.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:11.207+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:12.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:11.207+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:12.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.207+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:12.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.207+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:12.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.207+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:12.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.364+0000 7ffb7d3a2640 10 client.4483.objecter tick 2024-03-28T16:17:12.370 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.364+0000 7ffb7d3a2640 2 client.4483.objecter tid 2 on osd.6 is laggy 2024-03-28T16:17:12.371 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.364+0000 7ffb7d3a2640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:17:12.371 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.364+0000 7ffb7d3a2640 10 monclient: _renew_subs 2024-03-28T16:17:12.371 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.364+0000 7ffb7d3a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:17:12.371 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.364+0000 7ffb7d3a2640 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=44}) v3 -- 0x7ffb4c0038c0 con 0x55e93ad9ace0 2024-03-28T16:17:12.371 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.364+0000 7ffb7d3a2640 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- ping magic: 0 v1 -- 0x7ffb4c003b10 con 0x55e93acc8180 2024-03-28T16:17:12.371 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.368+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== mon.1 v2:172.21.15.83:3300/0 11 ==== osd_map(44..47 src has 1..47) v4 ==== 2206+0+0 (secure 0 0 0) 0x7ffb70054910 con 0x55e93ad9ace0 2024-03-28T16:17:12.371 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.368+0000 7ffb78939640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_map(44..47 src has 1..47) v4 2024-03-28T16:17:12.371 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.368+0000 7ffb78939640 3 client.4483.objecter handle_osd_map got epochs [44,47] > 43 2024-03-28T16:17:12.371 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.368+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 44 2024-03-28T16:17:12.371 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.368+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:12.371 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.368+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 44 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:12.371 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.368+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:12.372 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.368+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 45 2024-03-28T16:17:12.372 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.368+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:12.372 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.368+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 45 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:17.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.368+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:17.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.368+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 46 2024-03-28T16:17:17.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.368+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:17.365 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.368+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 46 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:17.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.368+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:17.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.368+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 47 2024-03-28T16:17:17.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.368+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:17.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.368+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 47 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:17.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.368+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:17.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.368+0000 7ffb78939640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-28T16:17:17.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:12.368+0000 7ffb78939640 20 client.4483.objecter 2 19.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2024-03-28T16:17:17.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:13.207+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:17.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:13.207+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:17.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:13.207+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:17.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:13.507+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 1 ==== osd_map(31..45 src has 1..45) v4 ==== 18150+0+0 (crc 0 0 0) 0x7ffb7400c5a0 con 0x55e93acc8180 2024-03-28T16:17:17.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:13.507+0000 7ffb78939640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_map(31..45 src has 1..45) v4 2024-03-28T16:17:17.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:13.507+0000 7ffb78939640 3 client.4483.objecter handle_osd_map ignoring epochs [31,45] <= 47 2024-03-28T16:17:17.366 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:13.507+0000 7ffb78939640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-28T16:17:17.367 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:13.507+0000 7ffb78939640 20 client.4483.objecter 2 19.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2024-03-28T16:17:17.367 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:14.207+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:17.367 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:14.207+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:17.367 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:14.207+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:17.367 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:15.207+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:17.367 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:15.207+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:17.367 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:15.207+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:17.367 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:16.207+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:17.367 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:16.207+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:17.367 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:16.207+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:17.367 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.208+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:17.367 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.208+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:17.367 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.208+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:17.367 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.365+0000 7ffb7d3a2640 10 client.4483.objecter tick 2024-03-28T16:17:17.368 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.365+0000 7ffb7d3a2640 2 client.4483.objecter tid 2 on osd.6 is laggy 2024-03-28T16:17:17.488 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.365+0000 7ffb7d3a2640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:17:17.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.365+0000 7ffb7d3a2640 10 monclient: _renew_subs 2024-03-28T16:17:17.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.365+0000 7ffb7d3a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:17:17.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.365+0000 7ffb7d3a2640 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=48}) v3 -- 0x7ffb4c003ca0 con 0x55e93ad9ace0 2024-03-28T16:17:17.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.365+0000 7ffb7d3a2640 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- ping magic: 0 v1 -- 0x7ffb4c003e30 con 0x55e93acc8180 2024-03-28T16:17:17.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== mon.1 v2:172.21.15.83:3300/0 12 ==== osd_map(48..52 src has 1..52) v4 ==== 3940+0+0 (secure 0 0 0) 0x7ffb70056760 con 0x55e93ad9ace0 2024-03-28T16:17:17.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_map(48..52 src has 1..52) v4 2024-03-28T16:17:17.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 3 client.4483.objecter handle_osd_map got epochs [48,52] > 47 2024-03-28T16:17:17.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 48 2024-03-28T16:17:17.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:17.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 48 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:17.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:17.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 49 2024-03-28T16:17:17.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:17.489 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 49 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:17.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:17.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 50 2024-03-28T16:17:17.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:17.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 50 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:17.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:17.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 51 2024-03-28T16:17:17.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:17.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 51 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:17.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:17.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 52 2024-03-28T16:17:17.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 10 client.4483.objecter checking op 2 2024-03-28T16:17:17.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 52 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:17.490 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:18.268 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-03-28T16:17:18.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [----------] Global test environment set-up. 2024-03-28T16:17:18.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-03-28T16:17:18.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-03-28T16:17:18.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5427 ms) 2024-03-28T16:17:18.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-03-28T16:17:18.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (260 ms) 2024-03-28T16:17:18.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-03-28T16:17:18.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20105 ms) 2024-03-28T16:17:18.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-03-28T16:17:18.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: attempt 0 of 20 2024-03-28T16:17:18.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5244 ms) 2024-03-28T16:17:18.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (31036 ms total) 2024-03-28T16:17:18.269 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: 2024-03-28T16:17:18.270 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [----------] Global test environment tear-down 2024-03-28T16:17:18.270 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (31036 ms total) 2024-03-28T16:17:18.270 INFO:tasks.workunit.client.0.smithi007.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-03-28T16:17:18.858 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-28T16:17:18.858 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:17.488+0000 7ffb78939640 20 client.4483.objecter 2 19.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2024-03-28T16:17:18.858 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.208+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:18.858 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.208+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:18.858 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.208+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:18.858 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7a93d640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 2 ==== osd_op_reply(2 exist [writefull 0~5] v45'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7ffb7400cae0 con 0x55e93acc8180 2024-03-28T16:17:18.858 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7a93d640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_op_reply(2 exist [writefull 0~5] v45'1 uv1 ondisk = 0) v8 2024-03-28T16:17:18.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7a93d640 10 client.4483.objecter in handle_osd_op_reply 2024-03-28T16:17:18.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7a93d640 7 client.4483.objecter handle_osd_op_reply 2 ondisk uv 1 in 19.1 attempt 0 2024-03-28T16:17:18.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7a93d640 10 client.4483.objecter op 0 rval 0 len 0 2024-03-28T16:17:18.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7a93d640 15 client.4483.objecter handle_osd_op_reply completed tid 2 2024-03-28T16:17:18.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7a93d640 15 client.4483.objecter _finish_op 2 2024-03-28T16:17:18.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7a93d640 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:18.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7a93d640 15 client.4483.objecter _session_op_remove 6 2 2024-03-28T16:17:18.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7a93d640 5 client.4483.objecter 0 in flight 2024-03-28T16:17:18.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-03-28T16:17:18.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 10 librados: Objecter returned from writefull r=0 2024-03-28T16:17:18.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 10 client.4483.objecter _op_submit op 0x55e93ad83060 2024-03-28T16:17:18.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 20 client.4483.objecter _calc_target epoch 52 base exist @19 precalc_pgid 0 pgid 0.0 is_read 2024-03-28T16:17:18.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:18.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 10 client.4483.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [6,1] primary 6 2024-03-28T16:17:18.859 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 20 client.4483.objecter _get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:18.860 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 10 client.4483.objecter _op_submit oid exist '@19' '@19' [read 0~256] tid 3 osd.6 2024-03-28T16:17:18.860 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 20 client.4483.objecter get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:18.860 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 15 client.4483.objecter _session_op_assign 6 3 2024-03-28T16:17:18.860 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 15 client.4483.objecter _send_op 3 to 19.1 on osd.6 2024-03-28T16:17:18.860 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- osd_op(unknown.0.0:3 19.1 19:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52) v8 -- 0x55e93ad92310 con 0x55e93acc8180 2024-03-28T16:17:18.860 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:18.860 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 5 client.4483.objecter 1 in flight 2024-03-28T16:17:19.248 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 10 client.4483.objecter _op_submit op 0x55e9 504 creating+peering 2024-03-28T16:17:19.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 270 unknown 2024-03-28T16:17:19.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 57 creating+activating 2024-03-28T16:17:19.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 50 active+clean 2024-03-28T16:17:19.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-03-28T16:17:19.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: io: 2024-03-28T16:17:19.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: client: 225 B/s wr, 0 op/s rd, 0 op/s wr 2024-03-28T16:17:19.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-03-28T16:17:19.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-03-28T16:17:19.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ OK ] LibRadosService.StatusFormat (6336 ms) 2024-03-28T16:17:19.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ RUN ] LibRadosService.Status 2024-03-28T16:17:19.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ OK ] LibRadosService.Status (20058 ms) 2024-03-28T16:17:19.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [----------] 4 tests from LibRadosService (32029 ms total) 2024-03-28T16:17:19.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: 2024-03-28T16:17:19.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [----------] Global test environment tear-down 2024-03-28T16:17:19.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [==========] 4 tests from 1 test suite ran. (32029 ms total) 2024-03-28T16:17:19.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_service: [ PASSED ] 4 tests. 2024-03-28T16:17:20.384 INFO:tasks.workunit.client.0.smithi007.stdout:3ad83550 2024-03-28T16:17:20.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 20 client.4483.objecter _calc_target epoch 52 base noexist @19 precalc_pgid 0 pgid 0.0 is_read 2024-03-28T16:17:20.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 20 client.4483.objecter _calc_target target noexist @19 -> pgid 19.a7bbaa1 2024-03-28T16:17:20.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 10 client.4483.objecter _calc_target raw pgid 19.a7bbaa1 -> actual 19.1 acting [6,1] primary 6 2024-03-28T16:17:20.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 20 client.4483.objecter _get_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:20.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 10 client.4483.objecter _op_submit oid noexist '@19' '@19' [read 0~256] tid 4 osd.6 2024-03-28T16:17:20.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 20 client.4483.objecter get_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:20.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 15 client.4483.objecter _session_op_assign 6 4 2024-03-28T16:17:20.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 15 client.4483.objecter _send_op 4 to 19.1 on osd.6 2024-03-28T16:17:20.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- osd_op(unknown.0.0:4 19.1 19:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52) v8 -- 0x55e93acc61b0 con 0x55e93acc8180 2024-03-28T16:17:20.384 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 5 2024-03-28T16:17:20.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:18.858+0000 7ffb7d63f780 5 client.4483.objecter 2 in flight 2024-03-28T16:17:20.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:19.209+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:20.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:19.209+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:20.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:19.209+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:20.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.209+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:20.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.209+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:20.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.209+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:20.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.209+0000 7ffb637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:17:20.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7a93d640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 4 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7ffb7400cae0 con 0x55e93acc8180 2024-03-28T16:17:20.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 3 ==== osd_map(53..53 src has 1..53) v4 ==== 1048+0+0 (crc 0 0 0) 0x7ffb7400cd30 con 0x55e93acc8180 2024-03-28T16:17:20.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7a93d640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-03-28T16:17:20.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7a93d640 10 client.4483.objecter in handle_osd_op_reply 2024-03-28T16:17:20.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb78939640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_map(53..53 src has 1..53) v4 2024-03-28T16:17:20.385 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7a93d640 7 client.4483.objecter handle_osd_op_reply 3 ondisk uv 1 in 19.1 attempt 0 2024-03-28T16:17:20.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7a93d640 10 client.4483.objecter op 0 rval 0 len 5 2024-03-28T16:17:20.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7a93d640 15 client.4483.objecter handle_osd_op_reply completed tid 3 2024-03-28T16:17:20.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7a93d640 15 client.4483.objecter _finish_op 3 2024-03-28T16:17:20.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7a93d640 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 5 2024-03-28T16:17:20.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7a93d640 15 client.4483.objecter _session_op_remove 6 3 2024-03-28T16:17:20.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb78939640 3 client.4483.objecter handle_osd_map got epochs [53,53] > 52 2024-03-28T16:17:20.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7a93d640 5 client.4483.objecter 1 in flight 2024-03-28T16:17:20.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 53 2024-03-28T16:17:20.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7a93d640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 5 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7ffb7400cae0 con 0x55e93acc8180 2024-03-28T16:17:20.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7a93d640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-03-28T16:17:20.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7a93d640 10 client.4483.objecter in handle_osd_op_reply 2024-03-28T16:17:20.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb78939640 10 client.4483.objecter checking op 4 2024-03-28T16:17:20.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 53 base noexist @19 precalc_pgid 0 pgid 0.0 is_read 2024-03-28T16:17:20.386 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb78939640 20 client.4483.objecter _calc_target target noexist @19 -> pgid 19.a7bbaa1 2024-03-28T16:17:20.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb78939640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-28T16:17:20.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb78939640 20 client.4483.objecter 4 19.a7bbaa1 osd.6 noexist [read 0~256] 2024-03-28T16:17:20.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7a93d640 7 client.4483.objecter handle_osd_op_reply 4 ondisk uv 0 in 19.1 attempt 0 2024-03-28T16:17:20.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7a93d640 10 client.4483.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-03-28T16:17:20.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7a93d640 10 client.4483.objecter op 0 rval 0 len 0 2024-03-28T16:17:20.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7a93d640 15 client.4483.objecter handle_osd_op_reply completed tid 4 2024-03-28T16:17:20.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7a93d640 15 client.4483.objecter _finish_op 4 2024-03-28T16:17:20.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7a93d640 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:20.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7a93d640 15 client.4483.objecter _session_op_remove 6 4 2024-03-28T16:17:20.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7a93d640 5 client.4483.objecter 0 in flight 2024-03-28T16:17:20.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (1526 ms) 2024-03-28T16:17:20.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-03-28T16:17:20.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 10 client.4483.objecter _op_submit op 0x55e93ad83060 2024-03-28T16:17:20.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 20 client.4483.objecter _calc_target epoch 53 base exist @19 precalc_pgid 0 pgid 0.0 is_read 2024-03-28T16:17:20.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:20.387 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 10 client.4483.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [6,1] primary 6 2024-03-28T16:17:20.388 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 20 client.4483.objecter _get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:20.388 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 10 client.4483.objecter _op_submit oid exist '@19' '@19' [read 0~256] tid 5 osd.6 2024-03-28T16:17:20.388 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 20 client.4483.objecter get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:20.388 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 15 client.4483.objecter _session_op_assign 6 5 2024-03-28T16:17:21.424 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 15 client.4483.objecter _send_op 5 to 19.1 on osd.6 2024-03-28T16:17:21.424 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- osd_op(unknown.0.0:5 19.1 19:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53) v8 -- 0x55e93ad83550 con 0x55e93acc8180 2024-03-28T16:17:21.424 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:21.424 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 5 client.4483.objecter 1 in flight 2024-03-28T16:17:21.424 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 10 client.4483.objecter _op_submit op 0x55e93acc6590 2024-03-28T16:17:21.424 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 20 client.4483.objecter _calc_target epoch 53 base noexist @19 precalc_pgid 0 pgid 0.0 is_read 2024-03-28T16:17:21.424 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 20 client.4483.objecter _calc_target target noexist @19 -> pgid 19.a7bbaa1 2024-03-28T16:17:21.424 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 10 client.4483.objecter _calc_target raw pgid 19.a7bbaa1 -> actual 19.1 acting [6,1] primary 6 2024-03-28T16:17:21.425 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 20 client.4483.objecter _get_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:21.425 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 10 client.4483.objecter _op_submit oid noexist '@19' '@19' [read 0~256] tid 6 osd.6 2024-03-28T16:17:21.425 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 20 client.4483.objecter get_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:21.425 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 15 client.4483.objecter _session_op_assign 6 6 2024-03-28T16:17:21.425 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 15 client.4483.objecter _send_op 6 to 19.1 on osd.6 2024-03-28T16:17:21.425 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- osd_op(unknown.0.0:6 19.1 19:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53) v8 -- 0x55e93acc6a80 con 0x55e93acc8180 2024-03-28T16:17:21.425 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 5 2024-03-28T16:17:21.425 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:20.383+0000 7ffb7d63f780 5 client.4483.objecter 2 in flight 2024-03-28T16:17:21.425 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.209+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:21.425 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.209+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:21.425 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.209+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:21.425 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.423+0000 7ffb7a93d640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 6 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7ffb7400cae0 con 0x55e93acc8180 2024-03-28T16:17:21.425 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.423+0000 7ffb7a93d640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-03-28T16:17:21.426 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.423+0000 7ffb7a93d640 10 client.4483.objecter in handle_osd_op_reply 2024-03-28T16:17:21.426 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.423+0000 7ffb7a93d640 7 client.4483.objecter handle_osd_op_reply 5 ondisk uv 1 in 19.1 attempt 0 2024-03-28T16:17:21.426 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.423+0000 7ffb7a93d640 10 client.4483.objecter op 0 rval 0 len 5 2024-03-28T16:17:21.426 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.423+0000 7ffb7a93d640 15 client.4483.objecter handle_osd_op_reply completed tid 5 2024-03-28T16:17:21.426 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.423+0000 7ffb7a93d640 15 client.4483.objecter _finish_op 5 2024-03-28T16:17:21.426 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.423+0000 7ffb7a93d640 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 5 2024-03-28T16:17:21.426 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.423+0000 7ffb7a93d640 15 client.4483.objecter _session_op_remove 6 5 2024-03-28T16:17:21.426 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.423+0000 7ffb7a93d640 5 client.4483.objecter 1 in flight 2024-03-28T16:17:21.426 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.423+0000 7ffb7a93d640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 7 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7ffb7400cae0 con 0x55e93acc8180 2024-03-28T16:17:21.426 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.423+0000 7ffb7a93d640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-03-28T16:17:21.426 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.423+0000 7ffb7a93d640 10 client.4483.objecter in handle_osd_op_reply 2024-03-28T16:17:21.426 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.423+0000 7ffb7a93d640 7 client.4483.objecter handle_osd_op_reply 6 ondisk uv 0 in 19.1 attempt 0 2024-03-28T16:17:21.426 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.423+0000 7ffb7a93d640 10 client.4483.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-03-28T16:17:21.426 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.423+0000 7ffb7a93d640 10 client.4483.objecter op 0 rval 0 len 0 2024-03-28T16:17:21.426 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.423+0000 7ffb7a93d640 15 client.4483.objecter handle_osd_op_reply completed tid 6 2024-03-28T16:17:21.427 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.423+0000 7ffb7a93d640 15 client.4483.objecter _finish_op 6 2024-03-28T16:17:21.427 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.423+0000 7ffb7a93d640 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:21.427 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.423+0000 7ffb7a93d640 15 client.4483.objecter _session_op_remove 6 6 2024-03-28T16:17:21.427 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.423+0000 7ffb7a93d640 5 client.4483.objecter 0 in flight 2024-03-28T16:17:21.427 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1040 ms) 2024-03-28T16:17:21.427 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-03-28T16:17:21.427 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 10 client.4483.objecter _op_submit op 0x55e93acc6590 2024-03-28T16:17:21.427 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 20 client.4483.objecter _calc_target epoch 53 base exist @19 precalc_pgid 0 pgid 0.0 is_read 2024-03-28T16:17:21.427 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:21.427 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 10 client.4483.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [6,1] primary 6 2024-03-28T16:17:21.427 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 20 client.4483.objecter _get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:21.427 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 10 client.4483.objecter _op_submit oid exist '@19' '@19' [read 0~256] tid 7 osd.6 2024-03-28T16:17:21.427 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 20 client.4483.objecter get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:21.427 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 15 client.4483.objecter _session_op_assign 6 7 2024-03-28T16:17:21.427 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 15 client.4483.objecter _send_op 7 to 19.1 on osd.6 2024-03-28T16:17:21.428 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- osd_op(unknown.0.0:7 19.1 19:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53) v8 -- 0x55e93ad83060 con 0x55e93acc8180 2024-03-28T16:17:21.428 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:21.428 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 5 client.4483.objecter 1 in flight 2024-03-28T16:17:21.428 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 10 client.4483.objecter _op_submit op 0x55e93ae89450 2024-03-28T16:17:21.428 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 20 client.4483.objecter _calc_target epoch 53 base noexist @19 precalc_pgid 0 pgid 0.0 is_read 2024-03-28T16:17:21.428 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 20 client.4483.objecter _calc_target target noexist @19 -> pgid 19.a7bbaa1 2024-03-28T16:17:21.608 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:17:21.598+0000 ea59640 -1 osd.4 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:2 23.1a 23:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-03-28T16:17:21.710 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 10 client.4483.objecter _calc_target raw pgid 19.a7bbaa1 -> actual 19.1 acting [6,1] primary 6 2024-03-28T16:17:21.710 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 20 client.4483.objecter _get_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:21.710 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 10 client.4483.objecter _op_submit oid noexist '@19' '@19' [read 0~256] tid 8 osd.6 2024-03-28T16:17:21.710 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 20 client.4483.objecter get_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:21.710 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 15 client.4483.objecter _session_op_assign 6 8 2024-03-28T16:17:21.711 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 15 client.4483.objecter _send_op 8 to 19.1 on osd.6 2024-03-28T16:17:21.711 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- osd_op(unknown.0.0:8 19.1 19:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53) v8 -- 0x55e93ae89940 con 0x55e93acc8180 2024-03-28T16:17:21.711 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 5 2024-03-28T16:17:21.711 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.424+0000 7ffb7d63f780 5 client.4483.objecter 2 in flight 2024-03-28T16:17:21.711 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 8 ==== osd_map(54..54 src has 1..54) v4 ==== 1402+0+0 (crc 0 0 0) 0x7ffb7401aa00 con 0x55e93acc8180 2024-03-28T16:17:21.711 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7a93d640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 9 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7ffb7400cae0 con 0x55e93acc8180 2024-03-28T16:17:21.711 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb78939640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_map(54..54 src has 1..54) v4 2024-03-28T16:17:21.711 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7a93d640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-03-28T16:17:21.711 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7a93d640 10 client.4483.objecter in handle_osd_op_reply 2024-03-28T16:17:21.711 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb78939640 3 client.4483.objecter handle_osd_map got epochs [54,54] > 53 2024-03-28T16:17:21.711 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 54 2024-03-28T16:17:21.711 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb78939640 10 client.4483.objecter checking op 7 2024-03-28T16:17:21.711 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 54 base exist @19 precalc_pgid 0 pgid 0.0 is_read 2024-03-28T16:17:21.711 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:21.712 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb78939640 10 client.4483.objecter checking op 8 2024-03-28T16:17:21.712 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 54 base noexist @19 precalc_pgid 0 pgid 0.0 is_read 2024-03-28T16:17:21.712 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb78939640 20 client.4483.objecter _calc_target target noexist @19 -> pgid 19.a7bbaa1 2024-03-28T16:17:21.712 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb78939640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-28T16:17:21.712 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb78939640 20 client.4483.objecter 7 19.d3a59201 osd.6 exist [read 0~256] 2024-03-28T16:17:21.712 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb78939640 20 client.4483.objecter 8 19.a7bbaa1 osd.6 noexist [read 0~256] 2024-03-28T16:17:21.712 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7a93d640 7 client.4483.objecter handle_osd_op_reply 7 ondisk uv 1 in 19.1 attempt 0 2024-03-28T16:17:21.712 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7a93d640 10 client.4483.objecter op 0 rval 0 len 5 2024-03-28T16:17:21.712 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7a93d640 15 client.4483.objecter handle_osd_op_reply completed tid 7 2024-03-28T16:17:21.712 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7a93d640 15 client.4483.objecter _finish_op 7 2024-03-28T16:17:21.712 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7a93d640 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 5 2024-03-28T16:17:21.712 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7a93d640 15 client.4483.objecter _session_op_remove 6 7 2024-03-28T16:17:21.712 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7a93d640 5 client.4483.objecter 1 in flight 2024-03-28T16:17:21.712 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7a93d640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 10 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7ffb7400cae0 con 0x55e93acc8180 2024-03-28T16:17:21.712 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7a93d640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-03-28T16:17:21.713 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7a93d640 10 client.4483.objecter in handle_osd_op_reply 2024-03-28T16:17:21.713 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7a93d640 7 client.4483.objecter handle_osd_op_reply 8 ondisk uv 0 in 19.1 attempt 0 2024-03-28T16:17:21.713 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7a93d640 10 client.4483.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-03-28T16:17:21.713 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7a93d640 10 client.4483.objecter op 0 rval 0 len 0 2024-03-28T16:17:21.713 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7a93d640 15 client.4483.objecter handle_osd_op_reply completed tid 8 2024-03-28T16:17:21.713 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7a93d640 15 client.4483.objecter _finish_op 8 2024-03-28T16:17:21.713 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7a93d640 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:21.713 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7a93d640 15 client.4483.objecter _session_op_remove 6 8 2024-03-28T16:17:21.713 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7a93d640 5 client.4483.objecter 0 in flight 2024-03-28T16:17:21.713 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (287 ms) 2024-03-28T16:17:21.713 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-03-28T16:17:21.713 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7d63f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-03-28T16:17:21.713 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7d63f780 20 librados: queue_aio_write 0x55e93ac74f80 completion 0x55e93acc6f80 write_seq 1 2024-03-28T16:17:21.713 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7d63f780 10 client.4483.objecter _op_submit op 0x55e93acc6590 2024-03-28T16:17:21.714 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7d63f780 20 client.4483.objecter _calc_target epoch 54 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:21.714 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7d63f780 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:21.714 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7d63f780 10 client.4483.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [6,1] primary 6 2024-03-28T16:17:21.714 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7d63f780 20 client.4483.objecter _get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:21.714 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7d63f780 10 client.4483.objecter _op_submit oid exist '@19' '@19' [write 0~5 in=5b] tid 9 osd.6 2024-03-28T16:17:21.714 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7d63f780 20 client.4483.objecter get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:21.714 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7d63f780 15 client.4483.objecter _session_op_assign 6 9 2024-03-28T16:17:21.714 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7d63f780 15 client.4483.objecter _send_op 9 to 19.1 on osd.6 2024-03-28T16:17:22.584 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:17:22.581+0000 ea59640 -1 osd.4 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:2 23.1a 23:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-03-28T16:17:23.333 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- osd_op(unknown.0.0:9 19.1 19:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54) v8 -- 0x55e93ae49dd0 con 0x55e93acc8180 2024-03-28T16:17:23.333 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7d63f780 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:23.333 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7d63f780 5 client.4483.objecter 1 in flight 2024-03-28T16:17:23.334 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:21.710+0000 7ffb7d63f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-03-28T16:17:23.334 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:22.209+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:23.334 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:22.209+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:23.334 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:22.209+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:23.334 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:22.366+0000 7ffb7d3a2640 10 client.4483.objecter tick 2024-03-28T16:17:23.334 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.209+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:23.334 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.209+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:23.334 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.209+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:23.334 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7a93d640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 11 ==== osd_op_reply(9 exist [write 0~5] v54'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7ffb7400cae0 con 0x55e93acc8180 2024-03-28T16:17:23.334 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7a93d640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_op_reply(9 exist [write 0~5] v54'2 uv2 ondisk = 0) v8 2024-03-28T16:17:23.334 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7a93d640 10 client.4483.objecter in handle_osd_op_reply 2024-03-28T16:17:23.334 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7a93d640 7 client.4483.objecter handle_osd_op_reply 9 ondisk uv 2 in 19.1 attempt 0 2024-03-28T16:17:23.335 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7a93d640 10 client.4483.objecter op 0 rval 0 len 0 2024-03-28T16:17:23.335 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7a93d640 15 client.4483.objecter handle_osd_op_reply completed tid 9 2024-03-28T16:17:23.335 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7a93d640 15 client.4483.objecter _finish_op 9 2024-03-28T16:17:23.335 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7a93d640 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:23.335 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (1622 ms) 2024-03-28T16:17:23.335 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7a93d640 15 client.4483.objecter _session_op_remove 6 9 2024-03-28T16:17:23.335 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7a93d640 5 client.4483.objecter 0 in flight 2024-03-28T16:17:23.335 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-03-28T16:17:23.335 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7a93d640 20 librados: complete_aio_write 0x55e93acc6f80 2024-03-28T16:17:23.335 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7d63f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-03-28T16:17:23.335 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7d63f780 20 librados: queue_aio_write 0x55e93ac74f80 completion 0x55e93ae4a180 write_seq 2 2024-03-28T16:17:23.335 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7d63f780 10 client.4483.objecter _op_submit op 0x55e93acc6590 2024-03-28T16:17:23.335 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7d63f780 20 client.4483.objecter _calc_target epoch 54 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:23.336 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7d63f780 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:23.336 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7d63f780 10 client.4483.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [6,1] primary 6 2024-03-28T16:17:23.336 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7d63f780 20 client.4483.objecter _get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:23.336 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7d63f780 10 client.4483.objecter _op_submit oid exist '@19' '@19' [write 0~5 in=5b] tid 10 osd.6 2024-03-28T16:17:24.677 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:17:24.641+0000 12a61640 -1 osd.5 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4446.0:2 8.16 8:6f1c4894:test-rados-api-smithi007-40817-3::testobj:head [write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:26.020 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:17:26.020+0000 12a61640 -1 osd.5 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4446.0:2 8.16 8:6f1c4894:test-rados-api-smithi007-40817-3::testobj:head [write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:27.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7d63f780 20 client.4483.objecter get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:27.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7d63f780 15 client.4483.objecter _session_op_assign 6 10 2024-03-28T16:17:27.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7d63f780 15 client.4483.objecter _send_op 10 to 19.1 on osd.6 2024-03-28T16:17:27.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- osd_op(unknown.0.0:10 19.1 19:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54) v8 -- 0x55e93ae49dd0 con 0x55e93acc8180 2024-03-28T16:17:27.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7d63f780 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:27.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7d63f780 5 client.4483.objecter 1 in flight 2024-03-28T16:17:27.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.333+0000 7ffb7d63f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-03-28T16:17:27.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.412+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 12 ==== osd_map(55..56 src has 1..56) v4 ==== 1623+0+0 (crc 0 0 0) 0x7ffb74004cc0 con 0x55e93acc8180 2024-03-28T16:17:27.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.412+0000 7ffb78939640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_map(55..56 src has 1..56) v4 2024-03-28T16:17:27.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.412+0000 7ffb78939640 3 client.4483.objecter handle_osd_map got epochs [55,56] > 54 2024-03-28T16:17:27.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.412+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 55 2024-03-28T16:17:27.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.412+0000 7ffb78939640 10 client.4483.objecter checking op 10 2024-03-28T16:17:27.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.412+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 55 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:27.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.412+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:27.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.412+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 56 2024-03-28T16:17:27.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.412+0000 7ffb78939640 10 client.4483.objecter checking op 10 2024-03-28T16:17:27.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.412+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 56 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:27.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.412+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:27.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.412+0000 7ffb78939640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-28T16:17:27.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:23.412+0000 7ffb78939640 20 client.4483.objecter 10 19.d3a59201 osd.6 exist [write 0~5 in=5b] 2024-03-28T16:17:27.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:24.209+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:27.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:24.209+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:27.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:24.209+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:27.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:25.209+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:27.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:25.209+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:27.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:25.209+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:27.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:26.209+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:27.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:26.209+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:27.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:26.209+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:27.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:27.209+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:27.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:27.209+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:27.286 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:17:27.189+0000 12a61640 -1 osd.5 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4446.0:2 8.16 8:6f1c4894:test-rados-api-smithi007-40817-3::testobj:head [write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:27.344 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-03-28T16:17:27.270+0000 ea59640 -1 osd.0 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4570.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e32) 2024-03-28T16:17:27.361 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:27.242+0000 ea59640 -1 osd.3 60 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22.3e8d6b28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:27.629 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:03:00.185 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:27.631 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:00.204 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:27.638 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:00.211 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:27.646 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:03:00.202 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:27.744 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:00.317 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:27.748 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:00.321 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:27.780 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:03:00.337 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:27.816 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:00.389 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:27.891 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:00.464 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:28.143 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:17:28.142+0000 12a61640 -1 osd.5 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4446.0:2 8.16 8:6f1c4894:test-rados-api-smithi007-40817-3::testobj:head [write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:28.575 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:28.392+0000 ea59640 -1 osd.3 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22.3e8d6b28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:28.702 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-03-28T16:17:28.647+0000 ea59640 -1 osd.0 60 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4570.0:3 25.14 25.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e32) 2024-03-28T16:17:29.664 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:29.534+0000 ea59640 -1 osd.3 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22.3e8d6b28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:30.109 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:30.099+0000 ea59640 -1 osd.2 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:30.288 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:27.209+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:30.288 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:27.367+0000 7ffb7d3a2640 10 client.4483.objecter tick 2024-03-28T16:17:30.288 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:28.209+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:30.288 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:28.209+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:30.288 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:28.209+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:30.288 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:29.209+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:30.288 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:29.209+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:30.288 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:29.209+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:30.288 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.209+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:30.288 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.209+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:30.288 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.210+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:30.289 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.210+0000 7ffb637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-28T16:17:30.289 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.287+0000 7ffb7a93d640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 13 ==== osd_op_reply(10 exist [write 0~5] v56'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7ffb7400cae0 con 0x55e93acc8180 2024-03-28T16:17:30.289 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.287+0000 7ffb7a93d640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_op_reply(10 exist [write 0~5] v56'3 uv3 ondisk = 0) v8 2024-03-28T16:17:30.289 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.287+0000 7ffb7a93d640 10 client.4483.objecter in handle_osd_op_reply 2024-03-28T16:17:30.289 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.287+0000 7ffb7a93d640 7 client.4483.objecter handle_osd_op_reply 10 ondisk uv 3 in 19.1 attempt 0 2024-03-28T16:17:30.289 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.287+0000 7ffb7a93d640 10 client.4483.objecter op 0 rval 0 len 0 2024-03-28T16:17:30.289 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.287+0000 7ffb7a93d640 15 client.4483.objecter handle_osd_op_reply completed tid 10 2024-03-28T16:17:30.289 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.287+0000 7ffb7a93d640 15 client.4483.objecter _finish_op 10 2024-03-28T16:17:30.289 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.287+0000 7ffb7a93d640 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:30.289 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.287+0000 7ffb7a93d640 15 client.4483.objecter _session_op_remove 6 10 2024-03-28T16:17:30.289 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.287+0000 7ffb7a93d640 5 client.4483.objecter 0 in flight 2024-03-28T16:17:30.289 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.287+0000 7ffb7a93d640 20 librados: complete_aio_write 0x55e93ae4a180 2024-03-28T16:17:30.289 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (6955 ms) 2024-03-28T16:17:30.289 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-03-28T16:17:30.289 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.288+0000 7ffb7d63f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-03-28T16:17:30.289 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.288+0000 7ffb7d63f780 20 librados: queue_aio_write 0x55e93ac74f80 completion 0x55e93ae49fb0 write_seq 3 2024-03-28T16:17:30.289 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.288+0000 7ffb7d63f780 10 client.4483.objecter _op_submit op 0x55e93acc6590 2024-03-28T16:17:30.290 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.288+0000 7ffb7d63f780 20 client.4483.objecter _calc_target epoch 56 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:30.290 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.288+0000 7ffb7d63f780 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:30.290 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.288+0000 7ffb7d63f780 10 client.4483.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [6,1] primary 6 2024-03-28T16:17:30.290 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.288+0000 7ffb7d63f780 20 client.4483.objecter _get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:30.316 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.288+0000 7ffb7d63f780 10 client.4483.objecter _op_submit oid exist '@19' '@19' [write 0~5 in=5b] tid 11 osd.6 2024-03-28T16:17:30.316 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.288+0000 7ffb7d63f780 20 client.4483.objecter get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:30.316 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.288+0000 7ffb7d63f780 15 client.4483.objecter _session_op_assign 6 11 2024-03-28T16:17:30.316 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.288+0000 7ffb7d63f780 15 client.4483.objecter _send_op 11 to 19.1 on osd.6 2024-03-28T16:17:30.316 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.288+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- osd_op(unknown.0.0:11 19.1 19:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) v8 -- 0x55e93ae6c890 con 0x55e93acc8180 2024-03-28T16:17:30.316 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.288+0000 7ffb7d63f780 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:30.316 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.288+0000 7ffb7d63f780 5 client.4483.objecter 1 in flight 2024-03-28T16:17:30.316 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.288+0000 7ffb7d63f780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-03-28T16:17:30.316 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.315+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 14 ==== osd_map(57..63 src has 1..63) v4 ==== 6534+0+0 (crc 0 0 0) 0x7ffb7401be10 con 0x55e93acc8180 2024-03-28T16:17:30.316 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.315+0000 7ffb78939640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_map(57..63 src has 1..63) v4 2024-03-28T16:17:30.316 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 3 client.4483.objecter handle_osd_map got epochs [57,63] > 56 2024-03-28T16:17:30.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 57 2024-03-28T16:17:30.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 10 client.4483.objecter checking op 11 2024-03-28T16:17:30.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 57 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:30.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:30.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 58 2024-03-28T16:17:30.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 10 client.4483.objecter checking op 11 2024-03-28T16:17:30.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 58 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:30.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:30.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 59 2024-03-28T16:17:30.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 10 client.4483.objecter checking op 11 2024-03-28T16:17:30.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 59 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:30.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:30.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 60 2024-03-28T16:17:30.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 10 client.4483.objecter checking op 11 2024-03-28T16:17:30.317 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 60 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:30.318 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:30.318 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 61 2024-03-28T16:17:30.318 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 10 client.4483.objecter checking op 11 2024-03-28T16:17:30.446 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 61 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:30.447 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:30.447 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 62 2024-03-28T16:17:30.447 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 10 client.4483.objecter checking op 11 2024-03-28T16:17:30.447 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 62 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:30.447 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:30.447 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 63 2024-03-28T16:17:30.447 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 10 client.4483.objecter checking op 11 2024-03-28T16:17:30.447 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 63 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:30.447 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:30.447 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:17:30.447 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 10 monclient: _renew_subs 2024-03-28T16:17:30.447 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:17:30.447 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=64}) v3 -- 0x7ffb6808b680 con 0x55e93ad9ace0 2024-03-28T16:17:30.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-28T16:17:30.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.316+0000 7ffb78939640 20 client.4483.objecter 11 19.d3a59201 osd.6 exist [write 0~5 in=5b] 2024-03-28T16:17:30.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.446+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== mon.1 v2:172.21.15.83:3300/0 13 ==== osd_map(64..64 src has 1..64) v4 ==== 2043+0+0 (secure 0 0 0) 0x7ffb70085e60 con 0x55e93ad9ace0 2024-03-28T16:17:30.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.446+0000 7ffb78939640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_map(64..64 src has 1..64) v4 2024-03-28T16:17:30.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.446+0000 7ffb78939640 3 client.4483.objecter handle_osd_map got epochs [64,64] > 63 2024-03-28T16:17:30.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.446+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 64 2024-03-28T16:17:30.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.446+0000 7ffb78939640 10 client.4483.objecter checking op 11 2024-03-28T16:17:30.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.446+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 64 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:30.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.446+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:30.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.446+0000 7ffb78939640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:17:30.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.446+0000 7ffb78939640 10 monclient: _renew_subs 2024-03-28T16:17:30.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.446+0000 7ffb78939640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:17:30.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.446+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=65}) v3 -- 0x7ffb6808be20 con 0x55e93ad9ace0 2024-03-28T16:17:30.448 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.446+0000 7ffb78939640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-28T16:17:30.896 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:30.882+0000 ea59640 -1 osd.3 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22.3e8d6b28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:31.608 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:31.427+0000 ea59640 -1 osd.2 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:32.053 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:31.933+0000 ea59640 -1 osd.3 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22.3e8d6b28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:32.602 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:32.548+0000 ea59640 -1 osd.2 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:32.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:30.446+0000 7ffb78939640 20 client.4483.objecter 11 19.d3a59201 osd.6 exist [write 0~5 in=5b] 2024-03-28T16:17:32.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:31.210+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:32.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:31.210+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:32.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:31.210+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:32.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:31.483+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== mon.1 v2:172.21.15.83:3300/0 14 ==== osd_map(65..65 src has 1..65) v4 ==== 2019+0+0 (secure 0 0 0) 0x7ffb70085e60 con 0x55e93ad9ace0 2024-03-28T16:17:32.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:31.483+0000 7ffb78939640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_map(65..65 src has 1..65) v4 2024-03-28T16:17:32.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:31.484+0000 7ffb78939640 3 client.4483.objecter handle_osd_map got epochs [65,65] > 64 2024-03-28T16:17:32.693 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:31.484+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 65 2024-03-28T16:17:32.693 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:31.484+0000 7ffb78939640 10 client.4483.objecter checking op 11 2024-03-28T16:17:32.693 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:31.484+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 65 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:32.693 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:31.484+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:32.693 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:31.484+0000 7ffb78939640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:17:32.693 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:31.484+0000 7ffb78939640 10 monclient: _renew_subs 2024-03-28T16:17:32.693 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:31.484+0000 7ffb78939640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:17:32.693 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:31.484+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=66}) v3 -- 0x7ffb6808a180 con 0x55e93ad9ace0 2024-03-28T16:17:32.693 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:31.484+0000 7ffb78939640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-28T16:17:32.693 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:31.484+0000 7ffb78939640 20 client.4483.objecter 11 19.d3a59201 osd.6 exist [write 0~5 in=5b] 2024-03-28T16:17:32.693 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:32.210+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:32.693 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:32.210+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:32.693 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:32.210+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:32.693 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:32.368+0000 7ffb7d3a2640 10 client.4483.objecter tick 2024-03-28T16:17:32.694 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:32.692+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== mon.1 v2:172.21.15.83:3300/0 15 ==== osd_map(66..66 src has 1..66) v4 ==== 304+0+0 (secure 0 0 0) 0x7ffb70085e60 con 0x55e93ad9ace0 2024-03-28T16:17:32.694 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:32.692+0000 7ffb78939640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_map(66..66 src has 1..66) v4 2024-03-28T16:17:32.694 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:32.692+0000 7ffb78939640 3 client.4483.objecter handle_osd_map got epochs [66,66] > 65 2024-03-28T16:17:32.694 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:32.692+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 66 2024-03-28T16:17:32.694 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:32.692+0000 7ffb78939640 10 client.4483.objecter checking op 11 2024-03-28T16:17:32.694 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:32.692+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 66 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:32.694 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:32.692+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:32.694 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:32.692+0000 7ffb78939640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-28T16:17:32.694 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:32.692+0000 7ffb78939640 10 monclient: _renew_subs 2024-03-28T16:17:32.734 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-0 api_pool: Running main() from gmock_main.cc 2024-03-28T16:17:32.734 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-03-28T16:17:32.735 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [----------] Global test environment set-up. 2024-03-28T16:17:32.735 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-03-28T16:17:32.735 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-03-28T16:17:32.735 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolList (4283 ms) 2024-03-28T16:17:32.735 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-03-28T16:17:32.735 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3984 ms) 2024-03-28T16:17:32.735 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-03-28T16:17:32.735 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3653 ms) 2024-03-28T16:17:32.735 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-03-28T16:17:32.735 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (4134 ms) 2024-03-28T16:17:32.735 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-03-28T16:17:32.735 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (4025 ms) 2024-03-28T16:17:32.735 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-03-28T16:17:32.735 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5926 ms) 2024-03-28T16:17:32.736 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-03-28T16:17:32.736 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5560 ms) 2024-03-28T16:17:32.736 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-03-28T16:17:32.736 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5143 ms) 2024-03-28T16:17:32.736 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-03-28T16:17:32.736 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8952 ms) 2024-03-28T16:17:32.736 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [----------] 9 tests from LibRadosPools (45660 ms total) 2024-03-28T16:17:32.736 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: 2024-03-28T16:17:32.736 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [----------] Global test environment tear-down 2024-03-28T16:17:32.736 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (45660 ms total) 2024-03-28T16:17:32.736 INFO:tasks.workunit.client.0.smithi007.stdout: api_pool: [ PASSED ] 9 tests. 2024-03-28T16:17:33.539 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:33.482+0000 ea59640 -1 osd.3 65 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22.3e8d6b28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:33.803 INFO:tasks.workunit.client.0.smithi007.stdout:3-28T16:17:32.692+0000 7ffb78939640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.83:3300/0 2024-03-28T16:17:33.803 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:32.692+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=67}) v3 -- 0x7ffb68072320 con 0x55e93ad9ace0 2024-03-28T16:17:33.803 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:32.692+0000 7ffb78939640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-28T16:17:33.804 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:32.692+0000 7ffb78939640 20 client.4483.objecter 11 19.d3a59201 osd.6 exist [write 0~5 in=5b] 2024-03-28T16:17:33.804 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.210+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:33.804 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.210+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:33.804 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.210+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:33.804 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.760+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== mon.1 v2:172.21.15.83:3300/0 16 ==== osd_map(67..67 src has 1..67) v4 ==== 296+0+0 (secure 0 0 0) 0x7ffb70085e60 con 0x55e93ad9ace0 2024-03-28T16:17:33.804 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.761+0000 7ffb78939640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_map(67..67 src has 1..67) v4 2024-03-28T16:17:33.805 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.761+0000 7ffb78939640 3 client.4483.objecter handle_osd_map got epochs [67,67] > 66 2024-03-28T16:17:33.805 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.761+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 67 2024-03-28T16:17:33.805 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.761+0000 7ffb78939640 10 client.4483.objecter checking op 11 2024-03-28T16:17:33.805 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.761+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 67 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:33.805 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.761+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:33.805 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.761+0000 7ffb78939640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-28T16:17:33.805 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.761+0000 7ffb78939640 20 client.4483.objecter 11 19.d3a59201 osd.6 exist [write 0~5 in=5b] 2024-03-28T16:17:33.805 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.802+0000 7ffb7a93d640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 15 ==== osd_op_reply(11 exist [write 0~5] v63'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7ffb7400cae0 con 0x55e93acc8180 2024-03-28T16:17:33.806 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.802+0000 7ffb7a93d640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_op_reply(11 exist [write 0~5] v63'4 uv4 ondisk = 0) v8 2024-03-28T16:17:33.806 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.802+0000 7ffb7a93d640 10 client.4483.objecter in handle_osd_op_reply 2024-03-28T16:17:33.806 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.802+0000 7ffb7a93d640 7 client.4483.objecter handle_osd_op_reply 11 ondisk uv 4 in 19.1 attempt 0 2024-03-28T16:17:33.806 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.802+0000 7ffb7a93d640 10 client.4483.objecter op 0 rval 0 len 0 2024-03-28T16:17:33.806 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.802+0000 7ffb7a93d640 15 client.4483.objecter handle_osd_op_reply completed tid 11 2024-03-28T16:17:33.806 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.802+0000 7ffb7a93d640 15 client.4483.objecter _finish_op 11 2024-03-28T16:17:33.806 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.802+0000 7ffb7a93d640 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:33.806 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.802+0000 7ffb7a93d640 15 client.4483.objecter _session_op_remove 6 11 2024-03-28T16:17:33.806 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.802+0000 7ffb7a93d640 5 client.4483.objecter 0 in flight 2024-03-28T16:17:33.806 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.802+0000 7ffb7a93d640 20 librados: complete_aio_write 0x55e93ae49fb0 2024-03-28T16:17:33.806 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (3515 ms) 2024-03-28T16:17:33.806 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-03-28T16:17:33.806 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 10 client.4483.objecter _op_submit op 0x55e93ae48dc0 2024-03-28T16:17:34.386 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:34.323+0000 ea59640 -1 osd.2 65 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:34.456 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 20 client.4483.objecter _calc_target epoch 67 base exist @19 precalc_pgid 0 pgid 0.0 is_read 2024-03-28T16:17:34.457 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:34.457 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 10 client.4483.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [6,1] primary 6 2024-03-28T16:17:34.457 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 20 client.4483.objecter _get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:34.457 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 10 client.4483.objecter _op_submit oid exist '@19' '@19' [read 0~0] tid 12 osd.6 2024-03-28T16:17:34.457 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 20 client.4483.objecter get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:34.457 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 15 client.4483.objecter _session_op_assign 6 12 2024-03-28T16:17:34.457 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 15 client.4483.objecter _send_op 12 to 19.1 on osd.6 2024-03-28T16:17:34.457 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- osd_op(unknown.0.0:12 19.1 19:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67) v8 -- 0x55e93ae492b0 con 0x55e93acc8180 2024-03-28T16:17:34.457 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:34.457 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 5 client.4483.objecter 1 in flight 2024-03-28T16:17:34.457 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 10 client.4483.objecter _op_submit op 0x55e93ae49660 2024-03-28T16:17:34.457 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 20 client.4483.objecter _calc_target epoch 67 base noexist @19 precalc_pgid 0 pgid 0.0 is_read 2024-03-28T16:17:34.457 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 20 client.4483.objecter _calc_target target noexist @19 -> pgid 19.a7bbaa1 2024-03-28T16:17:34.458 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 10 client.4483.objecter _calc_target raw pgid 19.a7bbaa1 -> actual 19.1 acting [6,1] primary 6 2024-03-28T16:17:34.458 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 20 client.4483.objecter _get_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:34.458 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 10 client.4483.objecter _op_submit oid noexist '@19' '@19' [read 0~0] tid 13 osd.6 2024-03-28T16:17:34.458 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 20 client.4483.objecter get_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:34.458 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 15 client.4483.objecter _session_op_assign 6 13 2024-03-28T16:17:34.458 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 15 client.4483.objecter _send_op 13 to 19.1 on osd.6 2024-03-28T16:17:34.458 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- osd_op(unknown.0.0:13 19.1 19:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67) v8 -- 0x55e93ae6ce30 con 0x55e93acc8180 2024-03-28T16:17:34.458 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 5 2024-03-28T16:17:34.458 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:33.803+0000 7ffb7d63f780 5 client.4483.objecter 2 in flight 2024-03-28T16:17:34.458 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.210+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:34.458 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.210+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:34.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.210+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:34.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7a93d640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 16 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7ffb7400cae0 con 0x55e93acc8180 2024-03-28T16:17:34.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7a93d640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-03-28T16:17:34.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7a93d640 10 client.4483.objecter in handle_osd_op_reply 2024-03-28T16:17:34.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7a93d640 7 client.4483.objecter handle_osd_op_reply 12 ondisk uv 4 in 19.1 attempt 0 2024-03-28T16:17:34.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7a93d640 10 client.4483.objecter op 0 rval 0 len 5 2024-03-28T16:17:34.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7a93d640 15 client.4483.objecter handle_osd_op_reply completed tid 12 2024-03-28T16:17:34.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7a93d640 15 client.4483.objecter _finish_op 12 2024-03-28T16:17:34.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7a93d640 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 5 2024-03-28T16:17:34.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7a93d640 15 client.4483.objecter _session_op_remove 6 12 2024-03-28T16:17:34.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7a93d640 5 client.4483.objecter 1 in flight 2024-03-28T16:17:34.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7a93d640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 17 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7ffb7400cae0 con 0x55e93acc8180 2024-03-28T16:17:34.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7a93d640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-03-28T16:17:34.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7a93d640 10 client.4483.objecter in handle_osd_op_reply 2024-03-28T16:17:34.459 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7a93d640 7 client.4483.objecter handle_osd_op_reply 13 ondisk uv 0 in 19.1 attempt 0 2024-03-28T16:17:34.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7a93d640 10 client.4483.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-03-28T16:17:34.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7a93d640 10 client.4483.objecter op 0 rval 0 len 0 2024-03-28T16:17:34.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7a93d640 15 client.4483.objecter handle_osd_op_reply completed tid 13 2024-03-28T16:17:34.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7a93d640 15 client.4483.objecter _finish_op 13 2024-03-28T16:17:34.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7a93d640 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:34.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7a93d640 15 client.4483.objecter _session_op_remove 6 13 2024-03-28T16:17:34.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7a93d640 5 client.4483.objecter 0 in flight 2024-03-28T16:17:34.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (654 ms) 2024-03-28T16:17:34.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-03-28T16:17:34.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 10 client.4483.objecter _op_submit op 0x55e93ae48dc0 2024-03-28T16:17:34.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 20 client.4483.objecter _calc_target epoch 67 base exist @19 precalc_pgid 0 pgid 0.0 is_read 2024-03-28T16:17:34.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:34.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 10 client.4483.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [6,1] primary 6 2024-03-28T16:17:34.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 20 client.4483.objecter _get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:34.460 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 10 client.4483.objecter _op_submit oid exist '@19' '@19' [read 0~0] tid 14 osd.6 2024-03-28T16:17:34.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 20 client.4483.objecter get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:34.461 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 15 client.4483.objecter _session_op_assign 6 14 2024-03-28T16:17:34.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 15 client.4483.objecter _send_op 14 to 19.1 on osd.6 2024-03-28T16:17:34.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- osd_op(unknown.0.0:14 19.1 19:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67) v8 -- 0x55e93ae492b0 con 0x55e93acc8180 2024-03-28T16:17:34.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:34.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 5 client.4483.objecter 1 in flight 2024-03-28T16:17:34.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 10 client.4483.objecter _op_submit op 0x55e93ae49820 2024-03-28T16:17:34.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 20 client.4483.objecter _calc_target epoch 67 base noexist @19 precalc_pgid 0 pgid 0.0 is_read 2024-03-28T16:17:34.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 20 client.4483.objecter _calc_target target noexist @19 -> pgid 19.a7bbaa1 2024-03-28T16:17:34.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 10 client.4483.objecter _calc_target raw pgid 19.a7bbaa1 -> actual 19.1 acting [6,1] primary 6 2024-03-28T16:17:34.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 20 client.4483.objecter _get_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:34.682 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 10 client.4483.objecter _op_submit oid noexist '@19' '@19' [read 0~0] tid 15 osd.6 2024-03-28T16:17:34.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 20 client.4483.objecter get_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:34.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 15 client.4483.objecter _session_op_assign 6 15 2024-03-28T16:17:34.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 15 client.4483.objecter _send_op 15 to 19.1 on osd.6 2024-03-28T16:17:34.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- osd_op(unknown.0.0:15 19.1 19:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67) v8 -- 0x55e93ae6ce30 con 0x55e93acc8180 2024-03-28T16:17:34.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 5 2024-03-28T16:17:34.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.456+0000 7ffb7d63f780 5 client.4483.objecter 2 in flight 2024-03-28T16:17:34.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (225 ms) 2024-03-28T16:17:34.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-03-28T16:17:34.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7a93d640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 18 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7ffb7400cae0 con 0x55e93acc8180 2024-03-28T16:17:34.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7a93d640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-03-28T16:17:34.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7a93d640 10 client.4483.objecter in handle_osd_op_reply 2024-03-28T16:17:34.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7a93d640 7 client.4483.objecter handle_osd_op_reply 14 ondisk uv 4 in 19.1 attempt 0 2024-03-28T16:17:34.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7a93d640 10 client.4483.objecter op 0 rval 0 len 5 2024-03-28T16:17:34.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7a93d640 15 client.4483.objecter handle_osd_op_reply completed tid 14 2024-03-28T16:17:34.683 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7a93d640 15 client.4483.objecter _finish_op 14 2024-03-28T16:17:34.684 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7a93d640 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 5 2024-03-28T16:17:34.684 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7a93d640 15 client.4483.objecter _session_op_remove 6 14 2024-03-28T16:17:34.684 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7a93d640 5 client.4483.objecter 1 in flight 2024-03-28T16:17:34.684 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7a93d640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 19 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7ffb7400cae0 con 0x55e93acc8180 2024-03-28T16:17:34.684 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7a93d640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-03-28T16:17:34.684 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7a93d640 10 client.4483.objecter in handle_osd_op_reply 2024-03-28T16:17:34.684 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7a93d640 7 client.4483.objecter handle_osd_op_reply 15 ondisk uv 0 in 19.1 attempt 0 2024-03-28T16:17:34.684 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7a93d640 10 client.4483.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-03-28T16:17:34.684 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7a93d640 10 client.4483.objecter op 0 rval 0 len 0 2024-03-28T16:17:34.684 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7a93d640 15 client.4483.objecter handle_osd_op_reply completed tid 15 2024-03-28T16:17:34.684 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7a93d640 15 client.4483.objecter _finish_op 15 2024-03-28T16:17:34.684 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7a93d640 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:34.684 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7a93d640 15 client.4483.objecter _session_op_remove 6 15 2024-03-28T16:17:34.685 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7a93d640 5 client.4483.objecter 0 in flight 2024-03-28T16:17:34.685 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 10 client.4483.objecter _op_submit op 0x55e93ae48dc0 2024-03-28T16:17:34.685 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 20 client.4483.objecter _calc_target epoch 67 base exist @19 precalc_pgid 0 pgid 0.0 is_read 2024-03-28T16:17:34.685 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:34.685 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 10 client.4483.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [6,1] primary 6 2024-03-28T16:17:34.685 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 20 client.4483.objecter _get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:34.685 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 10 client.4483.objecter _op_submit oid exist '@19' '@19' [read 0~0] tid 16 osd.6 2024-03-28T16:17:34.685 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 20 client.4483.objecter get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:34.685 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 15 client.4483.objecter _session_op_assign 6 16 2024-03-28T16:17:34.685 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 15 client.4483.objecter _send_op 16 to 19.1 on osd.6 2024-03-28T16:17:34.685 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- osd_op(unknown.0.0:16 19.1 19:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67) v8 -- 0x55e93ae492b0 con 0x55e93acc8180 2024-03-28T16:17:34.685 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:34.685 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 5 client.4483.objecter 1 in flight 2024-03-28T16:17:34.685 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 10 client.4483.objecter _op_submit op 0x55e93ae6c220 2024-03-28T16:17:34.686 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 20 client.4483.objecter _calc_target epoch 67 base noexist @19 precalc_pgid 0 pgid 0.0 is_read 2024-03-28T16:17:34.686 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 20 client.4483.objecter _calc_target target noexist @19 -> pgid 19.a7bbaa1 2024-03-28T16:17:34.686 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 10 client.4483.objecter _calc_target raw pgid 19.a7bbaa1 -> actual 19.1 acting [6,1] primary 6 2024-03-28T16:17:34.686 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 20 client.4483.objecter _get_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:34.766 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:34.651+0000 ea59640 -1 osd.3 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22.3e8d6b28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:34.787 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 10 client.4483.objecter _op_submit oid noexist '@19' '@19' [read 0~0] tid 17 osd.6 2024-03-28T16:17:34.787 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 20 client.4483.objecter get_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:34.788 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 15 client.4483.objecter _session_op_assign 6 17 2024-03-28T16:17:34.788 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 15 client.4483.objecter _send_op 17 to 19.1 on osd.6 2024-03-28T16:17:34.788 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- osd_op(unknown.0.0:17 19.1 19:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67) v8 -- 0x55e93ae6c710 con 0x55e93acc8180 2024-03-28T16:17:34.788 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 5 2024-03-28T16:17:34.788 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.681+0000 7ffb7d63f780 5 client.4483.objecter 2 in flight 2024-03-28T16:17:34.788 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.778+0000 7ffb7a93d640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 20 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7ffb7400cae0 con 0x55e93acc8180 2024-03-28T16:17:34.788 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.778+0000 7ffb7a93d640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-03-28T16:17:34.789 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.778+0000 7ffb7a93d640 10 client.4483.objecter in handle_osd_op_reply 2024-03-28T16:17:34.789 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.778+0000 7ffb7a93d640 7 client.4483.objecter handle_osd_op_reply 16 ondisk uv 4 in 19.1 attempt 0 2024-03-28T16:17:34.789 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.778+0000 7ffb7a93d640 10 client.4483.objecter op 0 rval 0 len 5 2024-03-28T16:17:34.789 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.778+0000 7ffb7a93d640 15 client.4483.objecter handle_osd_op_reply completed tid 16 2024-03-28T16:17:34.789 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.778+0000 7ffb7a93d640 15 client.4483.objecter _finish_op 16 2024-03-28T16:17:34.789 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.778+0000 7ffb7a93d640 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 5 2024-03-28T16:17:34.789 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.778+0000 7ffb7a93d640 15 client.4483.objecter _session_op_remove 6 16 2024-03-28T16:17:34.789 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.778+0000 7ffb7a93d640 5 client.4483.objecter 1 in flight 2024-03-28T16:17:34.790 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7a93d640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 21 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7ffb7400cae0 con 0x55e93acc8180 2024-03-28T16:17:34.790 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7a93d640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-03-28T16:17:34.790 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7a93d640 10 client.4483.objecter in handle_osd_op_reply 2024-03-28T16:17:34.790 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7a93d640 7 client.4483.objecter handle_osd_op_reply 17 ondisk uv 0 in 19.1 attempt 0 2024-03-28T16:17:34.790 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7a93d640 10 client.4483.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-03-28T16:17:34.790 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7a93d640 10 client.4483.objecter op 0 rval 0 len 0 2024-03-28T16:17:34.790 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7a93d640 15 client.4483.objecter handle_osd_op_reply completed tid 17 2024-03-28T16:17:34.791 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7a93d640 15 client.4483.objecter _finish_op 17 2024-03-28T16:17:34.791 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7a93d640 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:34.791 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7a93d640 15 client.4483.objecter _session_op_remove 6 17 2024-03-28T16:17:35.511 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:35.422+0000 ea59640 -1 osd.2 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:35.729 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7a93d640 5 client.4483.objecter 0 in flight 2024-03-28T16:17:35.729 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (106 ms) 2024-03-28T16:17:35.729 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-03-28T16:17:35.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7d63f780 20 librados: queue_aio_write 0x55e93ac74f80 completion 0x55e93ae6d080 write_seq 4 2024-03-28T16:17:35.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7d63f780 10 client.4483.objecter _op_submit op 0x55e93ae496d0 2024-03-28T16:17:35.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7d63f780 20 client.4483.objecter _calc_target epoch 67 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:35.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7d63f780 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:35.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7d63f780 10 client.4483.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [6,1] primary 6 2024-03-28T16:17:35.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7d63f780 20 client.4483.objecter _get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:35.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7d63f780 10 client.4483.objecter _op_submit oid exist '@19' '@19' [writefull 0~5 in=5b] tid 18 osd.6 2024-03-28T16:17:35.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7d63f780 20 client.4483.objecter get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:35.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7d63f780 15 client.4483.objecter _session_op_assign 6 18 2024-03-28T16:17:35.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7d63f780 15 client.4483.objecter _send_op 18 to 19.1 on osd.6 2024-03-28T16:17:35.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- osd_op(unknown.0.0:18 19.1 19:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67) v8 -- 0x55e93ae48dc0 con 0x55e93acc8180 2024-03-28T16:17:35.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7d63f780 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:35.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:34.787+0000 7ffb7d63f780 5 client.4483.objecter 1 in flight 2024-03-28T16:17:35.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.210+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:35.730 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.210+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:35.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.210+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:35.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7a93d640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 22 ==== osd_op_reply(18 exist [writefull 0~5] v67'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7ffb7400cae0 con 0x55e93acc8180 2024-03-28T16:17:35.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7a93d640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_op_reply(18 exist [writefull 0~5] v67'5 uv5 ondisk = 0) v8 2024-03-28T16:17:35.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7a93d640 10 client.4483.objecter in handle_osd_op_reply 2024-03-28T16:17:35.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (942 ms) 2024-03-28T16:17:35.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-03-28T16:17:35.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7a93d640 7 client.4483.objecter handle_osd_op_reply 18 ondisk uv 5 in 19.1 attempt 0 2024-03-28T16:17:35.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7a93d640 10 client.4483.objecter op 0 rval 0 len 0 2024-03-28T16:17:35.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7a93d640 15 client.4483.objecter handle_osd_op_reply completed tid 18 2024-03-28T16:17:35.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7a93d640 15 client.4483.objecter _finish_op 18 2024-03-28T16:17:35.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7a93d640 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:35.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7a93d640 15 client.4483.objecter _session_op_remove 6 18 2024-03-28T16:17:35.731 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7a93d640 5 client.4483.objecter 0 in flight 2024-03-28T16:17:36.113 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:36.030+0000 ea59640 -1 osd.3 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22.3e8d6b28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:36.960 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:36.920+0000 ea59640 -1 osd.2 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:37.113 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7a93d640 20 librados: complete_aio_write 0x55e93ae6d080 2024-03-28T16:17:37.113 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7d63f780 20 librados: queue_aio_write 0x55e93ac74f80 completion 0x55e93ae49170 write_seq 5 2024-03-28T16:17:37.113 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7d63f780 10 client.4483.objecter _op_submit op 0x55e93ae496d0 2024-03-28T16:17:37.113 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7d63f780 20 client.4483.objecter _calc_target epoch 67 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:37.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7d63f780 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:37.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7d63f780 10 client.4483.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [6,1] primary 6 2024-03-28T16:17:37.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7d63f780 20 client.4483.objecter _get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:37.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7d63f780 10 client.4483.objecter _op_submit oid exist '@19' '@19' [writefull 0~5 in=5b] tid 19 osd.6 2024-03-28T16:17:37.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7d63f780 20 client.4483.objecter get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:37.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7d63f780 15 client.4483.objecter _session_op_assign 6 19 2024-03-28T16:17:37.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7d63f780 15 client.4483.objecter _send_op 19 to 19.1 on osd.6 2024-03-28T16:17:37.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- osd_op(unknown.0.0:19 19.1 19:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67) v8 -- 0x55e93ae48dc0 con 0x55e93acc8180 2024-03-28T16:17:37.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7d63f780 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:37.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.729+0000 7ffb7d63f780 5 client.4483.objecter 1 in flight 2024-03-28T16:17:37.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.745+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 23 ==== osd_map(68..68 src has 1..68) v4 ==== 1037+0+0 (crc 0 0 0) 0x7ffb740054b0 con 0x55e93acc8180 2024-03-28T16:17:37.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.745+0000 7ffb78939640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_map(68..68 src has 1..68) v4 2024-03-28T16:17:37.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.745+0000 7ffb78939640 3 client.4483.objecter handle_osd_map got epochs [68,68] > 67 2024-03-28T16:17:37.114 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.745+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 68 2024-03-28T16:17:37.115 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.745+0000 7ffb78939640 10 client.4483.objecter checking op 19 2024-03-28T16:17:37.115 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.745+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 68 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:37.115 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.745+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:37.115 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.745+0000 7ffb78939640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-28T16:17:37.115 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:35.745+0000 7ffb78939640 20 client.4483.objecter 19 19.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2024-03-28T16:17:37.115 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:36.210+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:37.115 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:36.210+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:37.115 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:36.210+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:37.115 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7a93d640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 24 ==== osd_op_reply(19 exist [writefull 0~5] v68'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7ffb7400cae0 con 0x55e93acc8180 2024-03-28T16:17:37.129 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7a93d640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_op_reply(19 exist [writefull 0~5] v68'6 uv6 ondisk = 0) v8 2024-03-28T16:17:37.129 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7a93d640 10 client.4483.objecter in handle_osd_op_reply 2024-03-28T16:17:37.129 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7a93d640 7 client.4483.objecter handle_osd_op_reply 19 ondisk uv 6 in 19.1 attempt 0 2024-03-28T16:17:37.130 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7a93d640 10 client.4483.objecter op 0 rval 0 len 0 2024-03-28T16:17:37.130 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7a93d640 15 client.4483.objecter handle_osd_op_reply completed tid 19 2024-03-28T16:17:37.130 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7a93d640 15 client.4483.objecter _finish_op 19 2024-03-28T16:17:37.130 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7a93d640 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:37.130 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7a93d640 15 client.4483.objecter _session_op_remove 6 19 2024-03-28T16:17:37.130 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7a93d640 5 client.4483.objecter 0 in flight 2024-03-28T16:17:37.130 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7a93d640 20 librados: complete_aio_write 0x55e93ae49170 2024-03-28T16:17:37.130 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (1384 ms) 2024-03-28T16:17:37.130 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-03-28T16:17:37.130 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7d63f780 20 librados: queue_aio_write 0x55e93ac74f80 completion 0x55e93ae6cb40 write_seq 6 2024-03-28T16:17:37.130 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7d63f780 10 client.4483.objecter _op_submit op 0x55e93ae48dc0 2024-03-28T16:17:37.130 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7d63f780 20 client.4483.objecter _calc_target epoch 68 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:37.130 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7d63f780 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:37.130 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7d63f780 10 client.4483.objecter _calc_target raw pgid 19.d3a59201 -> actual 19.1 acting [6,1] primary 6 2024-03-28T16:17:37.131 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7d63f780 20 client.4483.objecter _get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:37.131 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7d63f780 10 client.4483.objecter _op_submit oid exist '@19' '@19' [writefull 0~5 in=5b] tid 20 osd.6 2024-03-28T16:17:37.131 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7d63f780 20 client.4483.objecter get_session s=0x55e93ac41950 osd=6 3 2024-03-28T16:17:37.131 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7d63f780 15 client.4483.objecter _session_op_assign 6 20 2024-03-28T16:17:37.131 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7d63f780 15 client.4483.objecter _send_op 20 to 19.1 on osd.6 2024-03-28T16:17:37.131 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 --> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] -- osd_op(unknown.0.0:20 19.1 19:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68) v8 -- 0x55e93ae492b0 con 0x55e93acc8180 2024-03-28T16:17:37.131 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7d63f780 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:37.131 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.113+0000 7ffb7d63f780 5 client.4483.objecter 1 in flight 2024-03-28T16:17:37.131 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.129+0000 7ffb78939640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 25 ==== osd_map(69..69 src has 1..69) v4 ==== 660+0+0 (crc 0 0 0) 0x7ffb74005340 con 0x55e93acc8180 2024-03-28T16:17:37.131 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.129+0000 7ffb78939640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_map(69..69 src has 1..69) v4 2024-03-28T16:17:37.131 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.129+0000 7ffb78939640 3 client.4483.objecter handle_osd_map got epochs [69,69] > 68 2024-03-28T16:17:37.131 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.129+0000 7ffb78939640 3 client.4483.objecter handle_osd_map decoding incremental epoch 69 2024-03-28T16:17:37.131 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.129+0000 7ffb78939640 10 client.4483.objecter checking op 20 2024-03-28T16:17:37.313 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:37.238+0000 ea59640 -1 osd.3 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22.3e8d6b28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:38.275 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:38.194+0000 ea59640 -1 osd.2 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:38.278 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:38.277+0000 ea59640 -1 osd.3 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22.3e8d6b28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:38.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.129+0000 7ffb78939640 20 client.4483.objecter _calc_target epoch 69 base exist @19 precalc_pgid 0 pgid 0.0 is_write 2024-03-28T16:17:38.523 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.129+0000 7ffb78939640 20 client.4483.objecter _calc_target target exist @19 -> pgid 19.d3a59201 2024-03-28T16:17:38.524 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.129+0000 7ffb78939640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-28T16:17:38.524 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.129+0000 7ffb78939640 20 client.4483.objecter 20 19.d3a59201 osd.6 exist [writefull 0~5 in=5b] 2024-03-28T16:17:38.524 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.210+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:38.524 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.210+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:38.524 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.210+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:38.524 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:37.369+0000 7ffb7d3a2640 10 client.4483.objecter tick 2024-03-28T16:17:38.524 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.210+0000 7ffb637fe640 10 monclient: tick 2024-03-28T16:17:38.524 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.210+0000 7ffb637fe640 10 monclient: _check_auth_tickets 2024-03-28T16:17:38.524 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.210+0000 7ffb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-28T16:17:38.524 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7a93d640 1 -- 172.21.15.7:0/3632289376 <== osd.6 v2:172.21.15.83:6810/35297 26 ==== osd_op_reply(20 exist [writefull 0~5] v69'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7ffb7400cae0 con 0x55e93acc8180 2024-03-28T16:17:38.524 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7a93d640 10 client.4483.objecter ms_dispatch 0x55e93ac6c060 osd_op_reply(20 exist [writefull 0~5] v69'7 uv7 ondisk = 0) v8 2024-03-28T16:17:38.524 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7a93d640 10 client.4483.objecter in handle_osd_op_reply 2024-03-28T16:17:38.524 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7a93d640 7 client.4483.objecter handle_osd_op_reply 20 ondisk uv 7 in 19.1 attempt 0 2024-03-28T16:17:38.525 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7a93d640 10 client.4483.objecter op 0 rval 0 len 0 2024-03-28T16:17:38.525 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7a93d640 15 client.4483.objecter handle_osd_op_reply completed tid 20 2024-03-28T16:17:38.525 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7a93d640 15 client.4483.objecter _finish_op 20 2024-03-28T16:17:38.525 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7a93d640 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 4 2024-03-28T16:17:38.525 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7a93d640 15 client.4483.objecter _session_op_remove 6 20 2024-03-28T16:17:38.525 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7a93d640 5 client.4483.objecter 0 in flight 2024-03-28T16:17:38.525 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7a93d640 20 librados: complete_aio_write 0x55e93ae6cb40 2024-03-28T16:17:38.525 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (1410 ms) 2024-03-28T16:17:38.525 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7d63f780 10 librados: watch_flush enter 2024-03-28T16:17:38.525 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7d63f780 10 librados: watch_flush exit 2024-03-28T16:17:38.525 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7d63f780 10 client.4483.objecter close_session for osd.6 2024-03-28T16:17:38.525 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 >> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] conn(0x55e93acc8180 msgr2=0x55e93ad82b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:17:38.525 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7d63f780 1 --2- 172.21.15.7:0/3632289376 >> [v2:172.21.15.83:6810/35297,v1:172.21.15.83:6811/35297] conn(0x55e93acc8180 0x55e93ad82b20 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:17:38.525 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7d63f780 20 client.4483.objecter put_session s=0x55e93ac41950 osd=6 1 2024-03-28T16:17:38.525 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7d63f780 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-28T16:17:38.526 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7d63f780 10 client.4483.objecter successfully canceled tick 2024-03-28T16:17:38.526 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7ffb6806a7b0 msgr2=0x7ffb6806cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:17:38.526 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7d63f780 1 --2- 172.21.15.7:0/3632289376 >> [v2:172.21.15.83:6800/34752,v1:172.21.15.83:6801/34752] conn(0x7ffb6806a7b0 0x7ffb6806cc70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c061000 tx=0x7ffb6c005ac0 comp rx=0 tx=0).stop 2024-03-28T16:17:38.526 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7d63f780 10 monclient: shutdown 2024-03-28T16:17:38.527 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7d63f780 20 monclient: shutdown discarding 0 pending message(s) 2024-03-28T16:17:38.527 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e93ad9ace0 msgr2=0x55e93ad90ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:17:38.527 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7d63f780 1 --2- 172.21.15.7:0/3632289376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e93ad9ace0 0x55e93ad90ad0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ffb7000aa90 tx=0x7ffb7000af50 comp rx=0 tx=0).stop 2024-03-28T16:17:38.527 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7b93f640 1 -- 172.21.15.7:0/3632289376 reap_dead start 2024-03-28T16:17:38.527 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 shutdown_connections 2024-03-28T16:17:38.527 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 >> 172.21.15.7:0/3632289376 conn(0x55e93ad08560 msgr2=0x55e93ad07230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:17:38.527 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 shutdown_connections 2024-03-28T16:17:38.527 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.523+0000 7ffb7d63f780 1 -- 172.21.15.7:0/3632289376 wait complete. 2024-03-28T16:17:38.527 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.524+0000 7ffb7d63f780 1 librados: shutdown 2024-03-28T16:17:38.527 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [----------] 12 tests from AsioRados (19666 ms total) 2024-03-28T16:17:38.527 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: 2024-03-28T16:17:38.527 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [----------] Global test environment tear-down 2024-03-28T16:17:38.527 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (51455 ms total) 2024-03-28T16:17:38.527 INFO:tasks.workunit.client.0.smithi007.stdout: api_asio: [ PASSED ] 12 tests. 2024-03-28T16:17:39.399 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:39.388+0000 ea59640 -1 osd.3 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22.3e8d6b28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:39.418 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:39.385+0000 ea59640 -1 osd.2 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:40.575 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:40.532+0000 ea59640 -1 osd.3 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22.3e8d6b28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:41.080 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:40.918+0000 ea59640 -1 osd.2 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:42.120 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:42.047+0000 ea59640 -1 osd.3 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22.3e8d6b28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:42.328 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:42.325+0000 ea59640 -1 osd.2 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:43.101 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:43.091+0000 ea59640 -1 osd.3 74 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22:14d6b17c:test-rados-api-smithi007-40576-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:43.322 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:43.320+0000 ea59640 -1 osd.2 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:44.341 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:44.339+0000 ea59640 -1 osd.2 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:45.641 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:45.620+0000 ea59640 -1 osd.2 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:45.804 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:45.803+0000 ea59640 -1 osd.3 77 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22:14d6b17c:test-rados-api-smithi007-40576-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:46.769 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:46.680+0000 ea59640 -1 osd.2 77 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:47.014 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:46.990+0000 ea59640 -1 osd.3 78 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22:14d6b17c:test-rados-api-smithi007-40576-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:48.121 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:20.694 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:48.128 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:20.701 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:48.144 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:48.070+0000 ea59640 -1 osd.3 79 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22:14d6b17c:test-rados-api-smithi007-40576-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:48.216 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:48.057+0000 ea59640 -1 osd.2 78 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:48.257 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:20.830 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:48.278 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:20.851 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:48.283 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:20.856 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:48.329 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:03:20.886 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:48.333 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:03:20.890 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:48.405 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:20.978 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:48.436 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:03:20.993 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:17:49.202 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:49.197+0000 ea59640 -1 osd.3 80 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22:14d6b17c:test-rados-api-smithi007-40576-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:49.412 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:49.410+0000 ea59640 -1 osd.2 79 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:50.253 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:50.246+0000 ea59640 -1 osd.3 81 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22:14d6b17c:test-rados-api-smithi007-40576-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:50.651 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:50.640+0000 ea59640 -1 osd.2 80 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:51.287 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:51.285+0000 ea59640 -1 osd.3 82 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22:14d6b17c:test-rados-api-smithi007-40576-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:51.690 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:51.688+0000 ea59640 -1 osd.2 81 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:52.345 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:52.338+0000 ea59640 -1 osd.3 83 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22:14d6b17c:test-rados-api-smithi007-40576-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:52.793 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:52.789+0000 ea59640 -1 osd.2 83 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:53.537 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:17:53.474+0000 ea59640 -1 osd.3 84 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4519.0:2 22.8 22:14d6b17c:test-rados-api-smithi007-40576-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:54.098 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:17:54.096+0000 ea59640 -1 osd.4 84 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:3 23.f 23:f4b50b61:::1.obj:head [write 0~172 in=172b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58) 2024-03-28T16:17:54.798 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-03-28T16:17:54.798 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-03-28T16:17:54.798 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-03-28T16:17:54.798 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-03-28T16:17:54.799 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (67530 ms) 2024-03-28T16:17:54.799 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (67530 ms total) 2024-03-28T16:17:54.799 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: 2024-03-28T16:17:54.799 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-03-28T16:17:54.799 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (67531 ms total) 2024-03-28T16:17:54.799 INFO:tasks.workunit.client.0.smithi007.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-03-28T16:17:55.105 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:17:55.102+0000 ea59640 -1 osd.4 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:3 23.f 23:f4b50b61:::1.obj:head [write 0~172 in=172b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58) 2024-03-28T16:17:55.286 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:55.192+0000 ea59640 -1 osd.2 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:56.100 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:17:56.092+0000 ea59640 -1 osd.4 86 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4543.0:3 23.f 23:f4b50b61:::1.obj:head [write 0~172 in=172b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58) 2024-03-28T16:17:56.193 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:56.170+0000 ea59640 -1 osd.2 86 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:57.113 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:17:57.105+0000 ea59640 -1 osd.4 87 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4543.0:3 23.f 23:f4b50b61:::1.obj:head [write 0~172 in=172b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58) 2024-03-28T16:17:57.172 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:57.153+0000 ea59640 -1 osd.2 86 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:58.114 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:58.112+0000 ea59640 -1 osd.2 87 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:58.135 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:17:58.133+0000 ea59640 -1 osd.4 88 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4543.0:3 23.f 23:f4b50b61:::1.obj:head [write 0~172 in=172b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58) 2024-03-28T16:17:59.086 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:17:59.084+0000 ea59640 -1 osd.2 88 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:17:59.222 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:17:59.206+0000 ea59640 -1 osd.4 89 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4334.0:20 3.2 3:441e7750:test-rados-api-smithi007-40188-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56) 2024-03-28T16:18:00.360 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:18:00.310+0000 ea59640 -1 osd.2 89 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:18:01.349 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:18:01.346+0000 ea59640 -1 osd.2 90 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:18:02.373 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:18:02.335+0000 ea59640 -1 osd.2 92 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:18:03.453 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:18:03.398+0000 ea59640 -1 osd.2 93 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:18:03.634 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:18:03.624+0000 12a61640 -1 osd.5 93 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4756.0:3 50.6 50:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65) 2024-03-28T16:18:04.502 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:18:04.427+0000 ea59640 -1 osd.2 93 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4507.0:2 18.1f 18:fc1ea691:test-rados-api-smithi007-40509-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-03-28T16:18:04.663 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:18:04.611+0000 12a61640 -1 osd.5 94 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4756.0:3 50.6 50:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65) 2024-03-28T16:18:05.656 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:18:05.656+0000 12a61640 -1 osd.5 94 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4756.0:3 50.6 50:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65) 2024-03-28T16:18:21.796 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-03-28T16:18:21.796 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-03-28T16:18:21.796 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-03-28T16:18:21.796 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-03-28T16:18:21.796 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-03-28T16:18:21.796 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-03-28T16:18:21.796 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-03-28T16:18:21.796 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (39038 ms) 2024-03-28T16:18:21.797 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-03-28T16:18:21.797 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (7253 ms) 2024-03-28T16:18:21.797 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-03-28T16:18:21.797 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (2881 ms) 2024-03-28T16:18:21.797 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-03-28T16:18:21.797 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (4043 ms) 2024-03-28T16:18:21.797 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-03-28T16:18:21.797 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (5012 ms) 2024-03-28T16:18:21.797 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-03-28T16:18:21.797 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (5295 ms) 2024-03-28T16:18:21.797 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-03-28T16:18:21.797 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (4313 ms) 2024-03-28T16:18:21.797 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-03-28T16:18:21.797 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (1315 ms) 2024-03-28T16:18:21.798 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-03-28T16:18:21.798 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (820 ms) 2024-03-28T16:18:21.798 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-03-28T16:18:21.798 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (2576 ms) 2024-03-28T16:18:21.798 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-03-28T16:18:21.798 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (10145 ms) 2024-03-28T16:18:21.798 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-03-28T16:18:21.798 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (414 ms) 2024-03-28T16:18:21.798 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-03-28T16:18:21.798 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (318 ms) 2024-03-28T16:18:21.798 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-03-28T16:18:21.798 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (1606 ms) 2024-03-28T16:18:21.798 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-03-28T16:18:21.799 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (289 ms) 2024-03-28T16:18:21.799 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-03-28T16:18:21.799 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (309 ms) 2024-03-28T16:18:21.799 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (85627 ms total) 2024-03-28T16:18:21.799 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: 2024-03-28T16:18:21.799 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-03-28T16:18:21.799 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (94553 ms total) 2024-03-28T16:18:21.799 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-03-28T16:18:25.327 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-03-28T16:18:25.327 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-03-28T16:18:25.327 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-03-28T16:18:25.327 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-03-28T16:18:25.327 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-03-28T16:18:25.327 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (970 ms) 2024-03-28T16:18:25.328 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-03-28T16:18:25.328 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (92 ms) 2024-03-28T16:18:25.328 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-03-28T16:18:25.328 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (97097 ms) 2024-03-28T16:18:25.328 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (98160 ms total) 2024-03-28T16:18:25.328 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: 2024-03-28T16:18:25.328 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-03-28T16:18:25.328 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (98161 ms total) 2024-03-28T16:18:25.328 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-03-28T16:18:26.101 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: Running main() from gmock_main.cc 2024-03-28T16:18:26.101 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-03-28T16:18:26.101 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [----------] Global test environment set-up. 2024-03-28T16:18:26.101 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-03-28T16:18:26.101 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-03-28T16:18:26.101 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (1006 ms) 2024-03-28T16:18:26.101 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-03-28T16:18:26.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (98 ms) 2024-03-28T16:18:26.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-03-28T16:18:26.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (93639 ms) 2024-03-28T16:18:26.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-03-28T16:18:26.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:15.725932+0000 mon.c [INF] from='client.? 172.21.15.7:0/171506667' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi007-40231-10"}]: dispatch 2024-03-28T16:18:26.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:15.745653+0000 mon.b [INF] from='client.? 172.21.15.7:0/422680187' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi007-40356-4", "tierpool": "test-rados-api-smithi007-40356-4-cache"}]: dispatch 2024-03-28T16:18:26.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:17.091096+0000 mon.c [INF] from='client.? 172.21.15.7:0/2456646845' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi007-40668-1"}]: dispatch 2024-03-28T16:18:26.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:17.372070+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi007-40356-4", "tierpool": "test-rados-api-smithi007-40356-4-cache"}]: dispatch 2024-03-28T16:18:26.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:17.574148+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi007-40231-10"}]: dispatch 2024-03-28T16:18:26.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:18.079248+0000 mon.a [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-28T16:18:26.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:18.952658+0000 mon.b [INF] from='client.? 172.21.15.7:0/974734387' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi007-40324-10"}]: dispatch 2024-03-28T16:18:26.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:19.113087+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi007-40668-1"}]: dispatch 2024-03-28T16:18:26.102 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:20.120570+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi007-40324-10"}]: dispatch 2024-03-28T16:18:26.103 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:18.839780+0000 mon.c [INF] from='client.? 172.21.15.7:0/171506667' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi007-40231-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T16:18:26.103 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:20.239376+0000 mon.a [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T16:18:26.103 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:20.670847+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi007-40231-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T16:18:26.103 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:20.785917+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi007-40356-4", "tierpool": "test-rados-api-smithi007-40356-4-cache"}]': finished 2024-03-28T16:18:26.103 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:20.882384+0000 mon.b [INF] from='client.? 172.21.15.7:0/4266172462' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi007-40545-7"}]: dispatch 2024-03-28T16:18:30.980 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:04:03.553 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:18:30.999 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:04:03.572 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:18:31.438 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:04:04.011 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:18:31.821 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:04:04.378 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:18:31.831 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:04:04.388 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:18:32.058 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:04:04.615 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:18:32.255 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:04:04.827 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:18:32.265 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:04:04.838 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:18:32.421 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:04:04.994 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:18:35.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-03-28T16:18:35.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [----------] Global test environment set-up. 2024-03-28T16:18:35.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [----------] 7 tests from LibRadosList 2024-03-28T16:18:35.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-03-28T16:18:35.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosList.ListObjects (85819 ms) 2024-03-28T16:18:35.690 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-03-28T16:18:35.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (1215 ms) 2024-03-28T16:18:35.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-03-28T16:18:35.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: myset foo1,foo2,foo3 2024-03-28T16:18:35.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo1 2024-03-28T16:18:35.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo2 2024-03-28T16:18:35.691 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo3 2024-03-28T16:18:35.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: myset foo1,foo4,foo5 2024-03-28T16:18:35.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo4 2024-03-28T16:18:35.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo5 2024-03-28T16:18:35.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo1 2024-03-28T16:18:35.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: myset foo6,foo7 2024-03-28T16:18:35.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo7 2024-03-28T16:18:35.692 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: foo6 2024-03-28T16:18:35.693 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-28T16:18:35.693 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns1:foo4 2024-03-28T16:18:35.693 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns1:foo5 2024-03-28T16:18:35.693 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns2:foo7 2024-03-28T16:18:35.693 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns2:foo6 2024-03-28T16:18:35.693 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: ns1:foo1 2024-03-28T16:18:35.694 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: :foo1 2024-03-28T16:18:35.694 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: :foo2 2024-03-28T16:18:35.694 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: :foo3 2024-03-28T16:18:35.694 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (9616 ms) 2024-03-28T16:18:35.694 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-03-28T16:18:35.694 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 1 0 2024-03-28T16:18:35.694 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 10 0 2024-03-28T16:18:35.695 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 13 0 2024-03-28T16:18:35.695 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 7 0 2024-03-28T16:18:35.695 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 14 0 2024-03-28T16:18:35.695 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 0 0 2024-03-28T16:18:35.695 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 15 0 2024-03-28T16:18:35.695 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 11 0 2024-03-28T16:18:35.695 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 5 0 2024-03-28T16:18:35.695 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 8 0 2024-03-28T16:18:35.695 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 6 0 2024-03-28T16:18:35.695 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 3 0 2024-03-28T16:18:35.695 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 4 0 2024-03-28T16:18:35.696 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 12 0 2024-03-28T16:18:35.696 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 9 0 2024-03-28T16:18:35.696 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: 2 0 2024-03-28T16:18:35.696 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-28T16:18:35.696 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (5326 ms) 2024-03-28T16:18:35.696 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-03-28T16:18:35.696 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: x cursor=MIN 2024-03-28T16:18:35.696 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=1 cursor=6:02547ec2:::1:head 2024-03-28T16:18:35.696 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=10 cursor=6:52ea6a34:::10:head 2024-03-28T16:18:35.696 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=13 cursor=6:566253c9:::13:head 2024-03-28T16:18:35.696 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=7 cursor=6:5c6b0b28:::7:head 2024-03-28T16:18:35.696 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=14 cursor=6:62a1935d:::14:head 2024-03-28T16:18:35.696 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=0 cursor=6:6cac518f:::0:head 2024-03-28T16:18:35.696 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=15 cursor=6:863748b0:::15:head 2024-03-28T16:18:35.696 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=11 cursor=6:89d3ae78:::11:head 2024-03-28T16:18:35.696 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=5 cursor=6:b29083e3:::5:head 2024-03-28T16:18:35.697 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=8 cursor=6:bd63b0f1:::8:head 2024-03-28T16:18:35.697 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=6 cursor=6:c4fdafeb:::6:head 2024-03-28T16:18:35.697 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=3 cursor=6:cfc208b3:::3:head 2024-03-28T16:18:35.697 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=4 cursor=6:d83876eb:::4:head 2024-03-28T16:18:35.697 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=12 cursor=6:de5d7c5f:::12:head 2024-03-28T16:18:35.697 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=9 cursor=6:e960b815:::9:head 2024-03-28T16:18:35.697 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > oid=2 cursor=6:f905c69b:::2:head 2024-03-28T16:18:35.697 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: FIRST> seek to MIN oid=1 2024-03-28T16:18:35.697 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=1 cursor=6:02547ec2:::1:head 2024-03-28T16:18:35.697 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:02547ec2:::1:head 2024-03-28T16:18:35.697 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:02547ec2:::1:head -> 1 2024-03-28T16:18:35.697 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=10 cursor=6:52ea6a34:::10:head 2024-03-28T16:18:35.697 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:52ea6a34:::10:head 2024-03-28T16:18:35.697 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:52ea6a34:::10:head -> 10 2024-03-28T16:18:35.697 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=13 cursor=6:566253c9:::13:head 2024-03-28T16:18:35.698 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:566253c9:::13:head 2024-03-28T16:18:35.698 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:566253c9:::13:head -> 13 2024-03-28T16:18:35.698 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=7 cursor=6:5c6b0b28:::7:head 2024-03-28T16:18:35.698 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:5c6b0b28:::7:head 2024-03-28T16:18:35.698 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:5c6b0b28:::7:head -> 7 2024-03-28T16:18:38.248 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=14 cursor=6:62a1935d:::14: api_cmd: got: 2024-03-28T16:18:21.850067+0000 mon.b [INF] from='client.? 172.21.15.7:0/974734387' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi007-40324-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T16:18:38.248 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:21.973364+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi007-40545-7"}]: dispatch 2024-03-28T16:18:38.248 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:22.719599+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi007-40324-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T16:18:38.248 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:23.742608+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi007-40668-1"}]': finished 2024-03-28T16:18:38.248 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:23.748726+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi007-40231-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-28T16:18:38.248 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:23.775077+0000 mon.c [INF] from='client.? 172.21.15.7:0/1276808165' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi007-40175-16"}]: dispatch 2024-03-28T16:18:38.248 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:24.077582+0000 mon.b [INF] from='client.? 172.21.15.7:0/3420036176' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-28T16:18:38.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:24.673801+0000 client.admin [INF] onexx 2024-03-28T16:18:38.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:24.682172+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-28T16:18:38.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:25.032644+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi007-40175-16"}]: dispatch 2024-03-28T16:18:38.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:27.363259+0000 mon.b [INF] from='client.? 172.21.15.7:0/974734387' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi007-40324-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi007-40324-10"}]: dispatch 2024-03-28T16:18:38.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:27.538328+0000 mon.b [INF] from='client.? 172.21.15.7:0/4266172462' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi007-40545-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T16:18:38.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:27.783638+0000 mon.c [INF] from='client.? 172.21.15.7:0/1276808165' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi007-40175-16"}]: dispatch 2024-03-28T16:18:38.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:27.915368+0000 mon.b [INF] from='client.? 172.21.15.7:0/1698809125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi007-40188-22"}]: dispatch 2024-03-28T16:18:38.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:28.079699+0000 mon.b [INF] from='client.? 172.21.15.7:0/3607276567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi007-40524-7"}]: dispatch 2024-03-28T16:18:38.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:29.314061+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi007-40175-16"}]: dispatch 2024-03-28T16:18:38.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:29.450580+0000 mon.b [INF] from='client.? 172.21.15.7:0/3420036176' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-28T16:18:38.952 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:29.538962+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi007-40668-1"}]': finished 2024-03-28T16:18:38.952 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:29.635508+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi007-40324-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi007-40324-10"}]: dispatch 2024-03-28T16:18:38.953 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:29.653670+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi007-40545-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T16:18:38.953 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:29.664838+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi007-40188-22"}]: dispatch 2024-03-28T16:18:38.953 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:29.672421+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi007-40524-7"}]: dispatch 2024-03-28T16:18:38.953 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:30.789311+0000 client.admin [INF] twoxx 2024-03-28T16:18:38.953 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:30.820832+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-28T16:18:38.953 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:30.979301+0000 mon.c [INF] from='client.? 172.21.15.7:0/1276808165' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi007-40175-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T16:18:38.953 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:31.074804+0000 mon.b [INF] from='client.? 172.21.15.7:0/1698809125' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi007-40188-22"}]: dispatch 2024-03-28T16:18:38.954 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:31.355544+0000 mon.b [INF] from='client.? 172.21.15.7:0/3281406142' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi007-40737-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-28T16:18:38.954 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:31.539056+0000 mon.b [INF] from='client.? 172.21.15.7:0/2967714943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi007-40164-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-28T16:18:38.954 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:31.987269+0000 mon.a [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T16:18:38.954 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:32.697853+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi007-40188-22"}]: dispatch 2024-03-28T16:18:38.954 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:32.729274+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi007-40737-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-28T16:18:38.954 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:32.741540+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi007-40164-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-28T16:18:38.955 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:33.249571+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi007-40175-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T16:18:45.067 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:33.391576+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi007-40231-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi007-40231-10"}]': finished 2024-03-28T16:18:45.067 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:33.411643+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi007-40545-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-28T16:18:45.067 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:33.415622+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi007-40524-7"}]': finished 2024-03-28T16:18:45.067 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:35.007621+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-28T16:18:45.067 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:34.638348+0000 mon.b [INF] from='client.? 172.21.15.7:0/3607276567' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi007-40524-7"}]: dispatch 2024-03-28T16:18:45.067 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:34.646802+0000 mon.b [INF] from='client.? 172.21.15.7:0/1698809125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi007-40188-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T16:18:45.067 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:34.693037+0000 mon.b [INF] from='client.? 172.21.15.7:0/4266172462' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi007-40545-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi007-40545-7"}]: dispatch 2024-03-28T16:18:45.067 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:36.270296+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi007-40524-7"}]: dispatch 2024-03-28T16:18:45.067 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:36.287338+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi007-40188-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T16:18:45.067 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:36.292400+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi007-40545-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi007-40545-7"}]: dispatch 2024-03-28T16:18:45.068 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:36.380389+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi007-40324-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi007-40324-10"}]': finished 2024-03-28T16:18:45.068 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:36.388182+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi007-40737-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-28T16:18:45.068 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:36.391214+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi007-40164-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-28T16:18:45.068 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:36.395512+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi007-40175-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-28T16:18:45.068 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:37.502507+0000 mon.a [INF] from='client.? 172.21.15.7:0/2348788166' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi007-40141-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-28T16:18:45.068 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:38.065617+0000 mon.b [INF] from='client.? 172.21.15.7:0/3420036176' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-28T16:18:45.417 INFO:tasks.workunit.client.0.smithi007.stdout: api api_stat: Running main() from gmock_main.cc 2024-03-28T16:18:45.417 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-03-28T16:18:45.417 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [----------] Global test environment set-up. 2024-03-28T16:18:45.417 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-03-28T16:18:45.417 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-03-28T16:18:45.417 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStat.Stat (33482 ms) 2024-03-28T16:18:45.418 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-03-28T16:18:45.418 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (6629 ms) 2024-03-28T16:18:45.418 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-03-28T16:18:45.418 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStat.StatNS (10766 ms) 2024-03-28T16:18:45.418 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-03-28T16:18:45.418 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (11 ms) 2024-03-28T16:18:45.418 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-03-28T16:18:45.418 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (2077 ms) 2024-03-28T16:18:45.418 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [----------] 5 tests from LibRadosStat (52965 ms total) 2024-03-28T16:18:45.418 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: 2024-03-28T16:18:45.418 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-03-28T16:18:45.418 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-03-28T16:18:45.418 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (6447 ms) 2024-03-28T16:18:45.419 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-03-28T16:18:45.419 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (21159 ms) 2024-03-28T16:18:45.419 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-03-28T16:18:45.419 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (1903 ms) 2024-03-28T16:18:45.419 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-03-28T16:18:45.419 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (1515 ms) 2024-03-28T16:18:45.419 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (31024 ms total) 2024-03-28T16:18:45.419 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: 2024-03-28T16:18:45.419 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [----------] Global test environment tear-down 2024-03-28T16:18:45.419 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (118308 ms total) 2024-03-28T16:18:45.419 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat: [ PASSED ] 9 tests. 2024-03-28T16:19:01.543 INFO:tasks.workunit.client.0.smithi007.stdout:_cmd: got: 2024-03-28T16:18:38.516875+0000 mon.a [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T16:19:01.651 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:38.263731+0000 mon.c [INF] from='client.? 172.21.15.7:0/1276808165' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi007-40175-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi007-40175-16"}]: dispatch 2024-03-28T16:19:01.651 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:39.082626+0000 client.admin [INF] threexx 2024-03-28T16:19:01.651 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:39.196642+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-28T16:19:01.651 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:39.759032+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi007-40175-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi007-40175-16"}]: dispatch 2024-03-28T16:19:01.651 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:39.889746+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi007-40524-7"}]': finished 2024-03-28T16:19:01.651 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:39.894993+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi007-40188-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-28T16:19:01.652 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:39.921042+0000 mon.a [INF] from='client.? 172.21.15.7:0/2348788166' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi007-40141-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-28T16:19:01.652 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: got: 2024-03-28T16:18:41.234537+0000 mon.a [INF] from='client.? 172.21.15.7:0/2335875654' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi007-40668-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-28T16:19:01.652 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (39643 ms) 2024-03-28T16:19:01.652 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (134386 ms total) 2024-03-28T16:19:01.652 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: 2024-03-28T16:19:01.652 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [----------] Global test environment tear-down 2024-03-28T16:19:01.652 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (134386 ms total) 2024-03-28T16:19:01.652 INFO:tasks.workunit.client.0.smithi007.stdout: api_cmd: [ PASSED ] 4 tests. 2024-03-28T16:19:02.020 INFO:tasks.workunit.client.0.smithi007.stdout:head 2024-03-28T16:19:02.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:62a1935d:::14:head 2024-03-28T16:19:02.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:62a1935d:::14:head -> 14 2024-03-28T16:19:02.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=0 cursor=6:6cac518f:::0:head 2024-03-28T16:19:02.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:6cac518f:::0:head 2024-03-28T16:19:02.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:6cac518f:::0:head -> 0 2024-03-28T16:19:02.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=15 cursor=6:863748b0:::15:head 2024-03-28T16:19:02.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:863748b0:::15:head 2024-03-28T16:19:02.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:863748b0:::15:head -> 15 2024-03-28T16:19:02.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=11 cursor=6:89d3ae78:::11:head 2024-03-28T16:19:02.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:89d3ae78:::11:head 2024-03-28T16:19:02.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:89d3ae78:::11:head -> 11 2024-03-28T16:19:02.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=5 cursor=6:b29083e3:::5:head 2024-03-28T16:19:02.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:b29083e3:::5:head 2024-03-28T16:19:02.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:b29083e3:::5:head -> 5 2024-03-28T16:19:02.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=8 cursor=6:bd63b0f1:::8:head 2024-03-28T16:19:02.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:bd63b0f1:::8:head 2024-03-28T16:19:02.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:bd63b0f1:::8:head -> 8 2024-03-28T16:19:02.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=6 cursor=6:c4fdafeb:::6:head 2024-03-28T16:19:02.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:c4fdafeb:::6:head 2024-03-28T16:19:02.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:c4fdafeb:::6:head -> 6 2024-03-28T16:19:02.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=3 cursor=6:cfc208b3:::3:head 2024-03-28T16:19:02.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:cfc208b3:::3:head 2024-03-28T16:19:02.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:cfc208b3:::3:head -> 3 2024-03-28T16:19:02.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=4 cursor=6:d83876eb:::4:head 2024-03-28T16:19:02.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:d83876eb:::4:head 2024-03-28T16:19:02.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:d83876eb:::4:head -> 4 2024-03-28T16:19:02.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=12 cursor=6:de5d7c5f:::12:head 2024-03-28T16:19:02.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:de5d7c5f:::12:head 2024-03-28T16:19:02.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:de5d7c5f:::12:head -> 12 2024-03-28T16:19:02.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=9 cursor=6:e960b815:::9:head 2024-03-28T16:19:02.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:e960b815:::9:head 2024-03-28T16:19:02.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:e960b815:::9:head -> 9 2024-03-28T16:19:02.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : oid=2 cursor=6:f905c69b:::2:head 2024-03-28T16:19:02.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:f905c69b:::2:head 2024-03-28T16:19:02.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:f905c69b:::2:head -> 2 2024-03-28T16:19:02.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:bd63b0f1:::8:head 2024-03-28T16:19:02.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=6:bd63b0f1:::8:head expected=6:bd63b0f1:::8:head 2024-03-28T16:19:02.022 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:bd63b0f1:::8:head -> 8 2024-03-28T16:19:02.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=8 expected=8 2024-03-28T16:19:02.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:b29083e3:::5:head 2024-03-28T16:19:02.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=6:b29083e3:::5:head expected=6:b29083e3:::5:head 2024-03-28T16:19:02.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:b29083e3:::5:head -> 5 2024-03-28T16:19:02.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=5 expected=5 2024-03-28T16:19:02.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:89d3ae78:::11:head 2024-03-28T16:19:02.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=6:89d3ae78:::11:head expected=6:89d3ae78:::11:head 2024-03-28T16:19:02.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:89d3ae78:::11:head -> 11 2024-03-28T16:19:02.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=11 expected=11 2024-03-28T16:19:02.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:863748b0:::15:head 2024-03-28T16:19:02.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=6:863748b0:::15:head expected=6:863748b0:::15:head 2024-03-28T16:19:02.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:863748b0:::15:head -> 15 2024-03-28T16:19:02.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=15 expected=15 2024-03-28T16:19:02.023 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:6cac518f:::0:head 2024-03-28T16:19:02.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=6:6cac518f:::0:head expected=6:6cac518f:::0:head 2024-03-28T16:19:02.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:6cac518f:::0:head -> 0 2024-03-28T16:19:02.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=0 expected=0 2024-03-28T16:19:02.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:566253c9:::13:head 2024-03-28T16:19:02.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=6:566253c9:::13:head expected=6:566253c9:::13:head 2024-03-28T16:19:02.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:566253c9:::13:head -> 13 2024-03-28T16:19:02.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=13 expected=13 2024-03-28T16:19:02.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:de5d7c5f:::12:head 2024-03-28T16:19:02.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=6:de5d7c5f:::12:head expected=6:de5d7c5f:::12:head 2024-03-28T16:19:02.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:de5d7c5f:::12:head -> 12 2024-03-28T16:19:02.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=12 expected=12 2024-03-28T16:19:02.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:d83876eb:::4:head 2024-03-28T16:19:02.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : cursor()=6:d83876eb:::4:head expected=6:d83876eb:::4:head 2024-03-28T16:19:02.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: > 6:d83876eb:::4:head -> 4 2024-03-28T16:19:02.024 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : entry=4 expected=4 2024-03-28T16:19:02.025 INFO:tasks.workunit.client.0.smithi007.stdout: api_list: : seek to 6:f905c69b:::2:head 2024-03-28T16:19:07.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: Running main() from gmock_main.cc 2024-03-28T16:19:07.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-03-28T16:19:07.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [----------] Global test environment set-up. 2024-03-28T16:19:07.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-03-28T16:19:07.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: seed 40188 2024-03-28T16:19:07.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-03-28T16:19:07.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-03-28T16:19:07.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-03-28T16:19:07.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (27512 ms) 2024-03-28T16:19:07.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-03-28T16:19:07.209 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (5680 ms) 2024-03-28T16:19:07.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-03-28T16:19:07.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (914 ms) 2024-03-28T16:19:07.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-03-28T16:19:07.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (34931 ms) 2024-03-28T16:19:07.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-03-28T16:19:07.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (2832 ms) 2024-03-28T16:19:07.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-03-28T16:19:07.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (5672 ms) 2024-03-28T16:19:07.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-03-28T16:19:07.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (69 ms) 2024-03-28T16:19:07.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-03-28T16:19:07.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (106 ms) 2024-03-28T16:19:07.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-03-28T16:19:07.210 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (5949 ms) 2024-03-28T16:19:07.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-03-28T16:19:07.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (439 ms) 2024-03-28T16:19:07.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-03-28T16:19:07.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (112 ms) 2024-03-28T16:19:07.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-03-28T16:19:07.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (95 ms) 2024-03-28T16:19:07.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-03-28T16:19:07.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (249 ms) 2024-03-28T16:19:07.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-03-28T16:19:07.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (135 ms) 2024-03-28T16:19:07.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-03-28T16:19:07.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (1744 ms) 2024-03-28T16:19:07.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-28T16:19:07.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (270 ms) 2024-03-28T16:19:07.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-03-28T16:19:07.211 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (102 ms) 2024-03-28T16:19:07.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-03-28T16:19:07.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (82 ms) 2024-03-28T16:19:07.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-03-28T16:19:07.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (73 ms) 2024-03-28T16:19:07.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (86967 ms total) 2024-03-28T16:19:07.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: 2024-03-28T16:19:07.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-03-28T16:19:07.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-03-28T16:19:07.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (5643 ms) 2024-03-28T16:19:07.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-03-28T16:19:07.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (788 ms) 2024-03-28T16:19:07.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-03-28T16:19:07.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (216 ms) 2024-03-28T16:19:07.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-03-28T16:19:07.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (380 ms) 2024-03-28T16:19:07.212 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-03-28T16:19:11.646 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:04:44.218 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:19:11.658 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:04:44.231 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:19:11.892 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:04:44.465 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:19:11.930 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:04:44.503 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:19:11.946 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:04:44.519 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:19:12.118 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:04:44.691 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:19:12.242 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:04:44.798 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:19:12.262 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:04:44.818 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:19:12.402 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 api_misc: [==========] Running 12 tests from 4 test suites. 2024-03-28T16:19:12.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] Global test environment set-up. 2024-03-28T16:19:12.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-03-28T16:19:12.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-03-28T16:19:12.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-03-28T16:19:12.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-28T16:19:12.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: 2024-03-28T16:19:12.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-03-28T16:19:12.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-03-28T16:19:12.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: unable to get monitor info from DNS SRV with service name: 2024-03-28T16:16:47.295+0000 7fb725037780 -1 failed for service _ceph-mon._tcp 2024-03-28T16:19:12.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: ceph-mon 2024-03-28T16:19:12.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: 2024-03-28T16:16:47.295+0000 7fb725037780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-03-28T16:19:12.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (265 ms) 2024-03-28T16:19:12.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-03-28T16:19:12.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2008 ms) 2024-03-28T16:19:12.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2273 ms total) 2024-03-28T16:19:12.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: 2024-03-28T16:19:12.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-03-28T16:19:12.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-03-28T16:19:12.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x5560b5b8a470 2024-03-28T16:19:12.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x5560b5d9c560 2024-03-28T16:19:12.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x5560b67a11c0 2024-03-28T16:19:12.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x5560b5d12050 2024-03-28T16:19:12.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x5560b67cd6d0 2024-03-28T16:19:12.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x5560b5f0b180 2024-03-28T16:19:12.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x5560b5f0bc20 2024-03-28T16:19:12.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x5560b5f0c710 2024-03-28T16:19:12.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x5560b5f0d1d0 2024-03-28T16:19:12.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x5560b5f0dc90 2024-03-28T16:19:12.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x5560b5f0cf40 2024-03-28T16:19:12.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 0x5560b5f0ec70 2024-03-28T16:19:12.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: started 12 aios 2024-03-28T16:19:12.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x5560b5b8a470 2024-03-28T16:19:12.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x5560b5d9c560 2024-03-28T16:19:12.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x5560b67a11c0 2024-03-28T16:19:12.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x5560b5d12050 2024-03-28T16:19:12.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x5560b67cd6d0 2024-03-28T16:19:12.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x5560b5f0b180 2024-03-28T16:19:12.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x5560b5f0bc20 2024-03-28T16:19:12.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x5560b5f0c710 2024-03-28T16:19:12.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x5560b5f0d1d0 2024-03-28T16:19:12.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x5560b5f0dc90 2024-03-28T16:19:12.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x5560b5f0cf40 2024-03-28T16:19:12.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: waiting 0x5560b5f0ec70 2024-03-28T16:19:12.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: done. 2024-03-28T16:19:12.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (25986 ms) 2024-03-28T16:19:12.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (25986 ms total) 2024-03-28T16:19:12.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: 2024-03-28T16:19:12.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-03-28T16:19:12.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-03-28T16:19:12.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-03-28T16:19:12.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-03-28T16:19:12.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.Exec (25571 ms) 2024-03-28T16:19:12.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-03-28T16:19:12.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (15310 ms) 2024-03-28T16:19:12.407 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-03-28T16:19:12.407 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (385 ms) 2024-03-28T16:19:12.407 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-03-28T16:19:12.407 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4485 ms) 2024-03-28T16:19:12.407 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-03-28T16:19:12.407 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-03-28T16:19:12.407 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-03-28T16:19:12.407 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-03-28T16:19:12.407 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-03-28T16:19:12.434 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:04:44.991 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:19:14.468 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (69131 ms) 2024-03-28T16:19:14.469 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] 8 tests from LibRadosMisc (114882 ms total) 2024-03-28T16:19:14.469 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: 2024-03-28T16:19:14.469 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [----------] Global test environment tear-down 2024-03-28T16:19:14.469 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (147424 ms total) 2024-03-28T16:19:14.469 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc: [ PASSED ] 12 tests. 2024-03-28T16:19:15.675 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: Running main() from gmock_main.cc 2024-03-28T16:19:15.675 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-03-28T16:19:15.676 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [----------] Global test environment set-up. 2024-03-28T16:19:15.676 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-03-28T16:19:15.676 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-03-28T16:19:15.676 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (18514 ms) 2024-03-28T16:19:15.676 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-03-28T16:19:15.676 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.LockShared (5315 ms) 2024-03-28T16:19:15.676 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-03-28T16:19:15.676 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (11476 ms) 2024-03-28T16:19:15.676 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-03-28T16:19:15.676 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (5225 ms) 2024-03-28T16:19:15.676 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-03-28T16:19:15.676 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (31341 ms) 2024-03-28T16:19:15.676 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-03-28T16:19:15.676 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.Unlock (1627 ms) 2024-03-28T16:19:15.676 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-03-28T16:19:15.677 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (245 ms) 2024-03-28T16:19:15.677 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-03-28T16:19:15.677 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (809 ms) 2024-03-28T16:19:15.677 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [----------] 8 tests from LibRadosLock (74552 ms total) 2024-03-28T16:19:15.677 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: 2024-03-28T16:19:15.677 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-03-28T16:19:15.677 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-03-28T16:19:15.677 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (22933 ms) 2024-03-28T16:19:15.677 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-03-28T16:19:15.677 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (257 ms) 2024-03-28T16:19:15.677 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-03-28T16:19:15.677 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1209 ms) 2024-03-28T16:19:15.677 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-03-28T16:19:15.677 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1802 ms) 2024-03-28T16:19:15.678 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-03-28T16:19:15.678 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (189 ms) 2024-03-28T16:19:15.678 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-03-28T16:19:15.678 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (457 ms) 2024-03-28T16:19:15.678 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-03-28T16:19:15.678 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (394 ms) 2024-03-28T16:19:15.678 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-03-28T16:19:15.678 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (198 ms) 2024-03-28T16:19:15.678 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (27439 ms total) 2024-03-28T16:19:15.678 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: 2024-03-28T16:19:15.678 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [----------] Global test environment tear-down 2024-03-28T16:19:15.678 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (148689 ms total) 2024-03-28T16:19:15.678 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock: [ PASSED ] 16 tests. 2024-03-28T16:19:15.705 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-03-28T16:19:15.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-03-28T16:19:15.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-03-28T16:19:15.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-03-28T16:19:15.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: seed 40324 2024-03-28T16:19:15.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-03-28T16:19:15.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (9343 ms) 2024-03-28T16:19:15.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-03-28T16:19:15.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (14788 ms) 2024-03-28T16:19:15.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-03-28T16:19:15.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (10796 ms) 2024-03-28T16:19:15.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-03-28T16:19:15.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (33930 ms) 2024-03-28T16:19:15.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-03-28T16:19:15.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (1515 ms) 2024-03-28T16:19:15.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-03-28T16:19:15.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (1585 ms) 2024-03-28T16:19:15.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-03-28T16:19:15.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (1636 ms) 2024-03-28T16:19:15.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-03-28T16:19:15.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (303 ms) 2024-03-28T16:19:15.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (73896 ms total) 2024-03-28T16:19:15.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: 2024-03-28T16:19:15.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-03-28T16:19:15.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-03-28T16:19:15.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (13410 ms) 2024-03-28T16:19:15.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-03-28T16:19:15.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (167 ms) 2024-03-28T16:19:15.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-03-28T16:19:15.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (2297 ms) 2024-03-28T16:19:15.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-03-28T16:19:15.708 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (6196 ms) 2024-03-28T16:19:15.708 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-03-28T16:19:15.708 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (145 ms) 2024-03-28T16:19:15.708 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-03-28T16:19:15.708 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (369 ms) 2024-03-28T16:19:15.708 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-03-28T16:19:15.708 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (321 ms) 2024-03-28T16:19:15.708 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-03-28T16:19:15.708 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (168 ms) 2024-03-28T16:19:15.708 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (23073 ms total) 2024-03-28T16:19:15.708 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: 2024-03-28T16:19:15.708 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-03-28T16:19:15.708 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (148679 ms total) 2024-03-28T16:19:15.708 INFO:tasks.workunit.client.0.smithi007.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-03-28T16:19:15.720 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-03-28T16:19:15.720 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-03-28T16:19:15.720 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-03-28T16:19:15.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-03-28T16:19:15.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: seed 40545 2024-03-28T16:19:15.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-03-28T16:19:15.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (17677 ms) 2024-03-28T16:19:15.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-03-28T16:19:15.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (43683 ms) 2024-03-28T16:19:15.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-03-28T16:19:15.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (30 ms) 2024-03-28T16:19:15.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-03-28T16:19:15.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (1184 ms) 2024-03-28T16:19:15.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-03-28T16:19:15.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (17128 ms) 2024-03-28T16:19:15.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (79702 ms total) 2024-03-28T16:19:15.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: 2024-03-28T16:19:15.721 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-03-28T16:19:15.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-03-28T16:19:15.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (6720 ms) 2024-03-28T16:19:15.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-03-28T16:19:15.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (1058 ms) 2024-03-28T16:19:15.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-03-28T16:19:15.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (1898 ms) 2024-03-28T16:19:15.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-03-28T16:19:15.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (6118 ms) 2024-03-28T16:19:15.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (15794 ms total) 2024-03-28T16:19:15.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: 2024-03-28T16:19:15.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-03-28T16:19:15.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (148600 ms total) 2024-03-28T16:19:15.722 INFO:tasks.workunit.client.0.smithi007.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-03-28T16:19:17.761 INFO:tasks.workunit.client.0.smithi007.stdout:(110 ms) 2024-03-28T16:19:17.761 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-03-28T16:19:17.761 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (167 ms) 2024-03-28T16:19:17.761 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-03-28T16:19:17.762 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (1048 ms) 2024-03-28T16:19:17.762 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-03-28T16:19:17.762 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (305 ms) 2024-03-28T16:19:17.762 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-03-28T16:19:17.762 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (452 ms) 2024-03-28T16:19:17.762 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-03-28T16:19:17.762 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (166 ms) 2024-03-28T16:19:17.762 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-03-28T16:19:17.762 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (349 ms) 2024-03-28T16:19:17.762 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-03-28T16:19:17.762 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (166 ms) 2024-03-28T16:19:17.762 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-03-28T16:19:17.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (732 ms) 2024-03-28T16:19:17.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-03-28T16:19:17.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (251 ms) 2024-03-28T16:19:17.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-03-28T16:19:17.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (2010 ms) 2024-03-28T16:19:17.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-03-28T16:19:17.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (1102 ms) 2024-03-28T16:19:17.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-03-28T16:19:17.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (995 ms) 2024-03-28T16:19:17.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (14880 ms total) 2024-03-28T16:19:17.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: 2024-03-28T16:19:17.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [----------] Global test environment tear-down 2024-03-28T16:19:17.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (150798 ms total) 2024-03-28T16:19:17.763 INFO:tasks.workunit.client.0.smithi007.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-03-28T16:19:20.109 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-03-28T16:19:20.109 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-03-28T16:19:20.109 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-03-28T16:19:20.109 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-03-28T16:19:20.109 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-03-28T16:19:20.109 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-03-28T16:19:20.109 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-03-28T16:19:20.109 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: 2024-03-28T16:19:20.109 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-03-28T16:19:20.109 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-03-28T16:19:20.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (14059 ms) 2024-03-28T16:19:20.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-03-28T16:19:20.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (17443 ms) 2024-03-28T16:19:20.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-03-28T16:19:20.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (33928 ms) 2024-03-28T16:19:20.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-03-28T16:19:20.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (29249 ms) 2024-03-28T16:19:20.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-03-28T16:19:20.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (33318 ms) 2024-03-28T16:19:20.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-03-28T16:19:20.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (19430 ms) 2024-03-28T16:19:20.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-03-28T16:19:20.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (5478 ms) 2024-03-28T16:19:20.110 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (152905 ms total) 2024-03-28T16:19:20.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: 2024-03-28T16:19:20.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-03-28T16:19:20.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (152905 ms total) 2024-03-28T16:19:20.111 INFO:tasks.workunit.client.0.smithi007.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-03-28T16:19:20.116 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_1_[40893]: starting. 2024-03-28T16:19:20.116 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_1_[40893]: creating pool ceph_test_rados_open_pools_parallel.smithi007-40854 2024-03-28T16:19:20.116 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_1_[40893]: created object 0... 2024-03-28T16:19:20.116 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_1_[40893]: created object 25... 2024-03-28T16:19:20.116 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_1_[40893]: created object 49... 2024-03-28T16:19:20.116 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_1_[40893]: finishing. 2024-03-28T16:19:20.116 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_1_[40893]: shutting down. 2024-03-28T16:19:20.116 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_2_[40894]: starting. 2024-03-28T16:19:20.116 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_2_[40894]: rados_pool_create. 2024-03-28T16:19:20.116 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_2_[40894]: rados_ioctx_create. 2024-03-28T16:19:20.117 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_2_[40894]: shutting down. 2024-03-28T16:19:20.117 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: ******************************* 2024-03-28T16:19:20.117 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_3_[64753]: starting. 2024-03-28T16:19:20.117 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_3_[64753]: creating pool ceph_test_rados_open_pools_parallel.smithi007-40854 2024-03-28T16:19:20.117 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_3_[64753]: created object 0... 2024-03-28T16:19:20.117 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_3_[64753]: created object 25... 2024-03-28T16:19:20.117 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_3_[64753]: created object 49... 2024-03-28T16:19:20.117 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_3_[64753]: finishing. 2024-03-28T16:19:20.117 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_3_[64753]: shutting down. 2024-03-28T16:19:20.117 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: ******************************* 2024-03-28T16:19:20.117 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_4_[64754]: starting. 2024-03-28T16:19:20.117 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_4_[64754]: rados_pool_create. 2024-03-28T16:19:20.117 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_4_[64754]: rados_ioctx_create. 2024-03-28T16:19:20.117 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: process_4_[64754]: shutting down. 2024-03-28T16:19:20.118 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: ******************************* 2024-03-28T16:19:20.118 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: ******************************* 2024-03-28T16:19:20.118 INFO:tasks.workunit.client.0.smithi007.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-03-28T16:19:23.243 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-03-28T16:19:23.244 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-03-28T16:19:23.244 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-03-28T16:19:23.244 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-03-28T16:19:23.244 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-03-28T16:19:23.244 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify_test_cb 2024-03-28T16:19:23.244 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (82428 ms) 2024-03-28T16:19:23.244 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-03-28T16:19:23.244 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-28T16:19:23.244 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94358530293296 err -107 2024-03-28T16:19:23.244 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (2131 ms) 2024-03-28T16:19:23.244 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-03-28T16:19:23.244 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-28T16:19:23.244 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94358530295504 err -107 2024-03-28T16:19:23.244 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1859 ms) 2024-03-28T16:19:23.245 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-03-28T16:19:23.245 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4519 notify_id 420906795009 cookie 94358530320272 2024-03-28T16:19:23.245 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (2525 ms) 2024-03-28T16:19:23.245 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-03-28T16:19:23.245 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4519 notify_id 425201762304 cookie 94358530330848 2024-03-28T16:19:23.245 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (395 ms) 2024-03-28T16:19:23.245 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-03-28T16:19:23.245 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4519 notify_id 425201762304 cookie 94358530334256 2024-03-28T16:19:23.245 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (307 ms) 2024-03-28T16:19:23.245 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-03-28T16:19:23.245 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4519 notify_id 429496729600 cookie 94358530334256 2024-03-28T16:19:23.245 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4519 notify_id 429496729600 cookie 94358530349408 2024-03-28T16:19:23.245 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (2722 ms) 2024-03-28T16:19:23.245 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-03-28T16:19:23.245 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4519 notify_id 433791696897 cookie 94358530334256 2024-03-28T16:19:23.246 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4519 notify_id 433791696898 cookie 94358530334256 2024-03-28T16:19:23.246 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (5298 ms) 2024-03-28T16:19:23.246 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-03-28T16:19:23.246 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-03-28T16:19:23.246 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94358530334256 err -107 2024-03-28T16:19:23.246 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_cb from 4519 notify_id 446676598787 cookie 94358530334256 2024-03-28T16:19:23.246 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (6190 ms) 2024-03-28T16:19:23.246 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-03-28T16:19:23.246 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-03-28T16:19:23.246 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94358530334256 err -107 2024-03-28T16:19:23.246 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1284 ms) 2024-03-28T16:19:23.246 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (105139 ms total) 2024-03-28T16:19:23.246 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: 2024-03-28T16:19:23.246 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-03-28T16:19:23.246 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-03-28T16:19:23.247 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: watch_notify_test_cb 2024-03-28T16:19:23.247 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (5749 ms) 2024-03-28T16:19:23.247 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (5749 ms total) 2024-03-28T16:19:23.247 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: 2024-03-28T16:19:23.247 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-03-28T16:19:23.247 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (156102 ms total) 2024-03-28T16:19:23.247 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-03-28T16:19:24.270 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_1_[40868]: starting. 2024-03-28T16:19:24.271 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_1_[40868]: creating pool ceph_test_rados_delete_pools_parallel.smithi007-40844 2024-03-28T16:19:24.271 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_1_[40868]: created object 0... 2024-03-28T16:19:24.271 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_1_[40868]: created object 25... 2024-03-28T16:19:24.271 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_1_[40868]: created object 49... 2024-03-28T16:19:24.271 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_1_[40868]: finishing. 2024-03-28T16:19:24.271 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_1_[40868]: shutting down. 2024-03-28T16:19:24.271 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_2_[40874]: starting. 2024-03-28T16:19:24.271 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_2_[40874]: deleting pool ceph_test_rados_delete_pools_parallel.smithi007-40844 2024-03-28T16:19:24.271 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_2_[40874]: shutting down. 2024-03-28T16:19:24.271 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: ******************************* 2024-03-28T16:19:24.271 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_3_[64312]: starting. 2024-03-28T16:19:24.271 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_3_[64312]: creating pool ceph_test_rados_delete_pools_parallel.smithi007-40844 2024-03-28T16:19:24.271 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_3_[64312]: created object 0... 2024-03-28T16:19:24.271 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_3_[64312]: created object 25... 2024-03-28T16:19:24.271 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_3_[64312]: created object 49... 2024-03-28T16:19:24.271 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_3_[64312]: finishing. 2024-03-28T16:19:24.272 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_3_[64312]: shutting down. 2024-03-28T16:19:24.272 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: ******************************* 2024-03-28T16:19:24.272 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_5_[64314]: starting. 2024-03-28T16:19:24.272 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_5_[64314]: listing objects. 2024-03-28T16:19:24.272 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_5_[64314]: listed object 0... 2024-03-28T16:19:24.272 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_5_[64314]: listed object 25... 2024-03-28T16:19:24.272 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_5_[64314]: saw 50 objects 2024-03-28T16:19:24.272 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_5_[64314]: shutting down. 2024-03-28T16:19:24.272 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: ******************************* 2024-03-28T16:19:24.272 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_4_[64313]: starting. 2024-03-28T16:19:24.272 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_4_[64313]: deleting pool ceph_test_rados_delete_pools_parallel.smithi007-40844 2024-03-28T16:19:24.272 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: process_4_[64313]: shutting down. 2024-03-28T16:19:24.272 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: ******************************* 2024-03-28T16:19:24.272 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: ******************************* 2024-03-28T16:19:24.272 INFO:tasks.workunit.client.0.smithi007.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-03-28T16:19:27.862 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: Running main() from gmock_main.cc 2024-03-28T16:19:27.862 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-03-28T16:19:27.862 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [----------] Global test environment set-up. 2024-03-28T16:19:27.862 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-03-28T16:19:27.862 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-03-28T16:19:27.862 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (16585 ms) 2024-03-28T16:19:27.862 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-03-28T16:19:27.862 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-03-28T16:19:27.862 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-03-28T16:19:27.862 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (16491 ms) 2024-03-28T16:19:27.863 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-03-28T16:19:27.863 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (4519 ms) 2024-03-28T16:19:27.863 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-03-28T16:19:27.863 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.Checksum (34146 ms) 2024-03-28T16:19:27.863 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-03-28T16:19:27.863 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (1442 ms) 2024-03-28T16:19:27.863 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-03-28T16:19:27.863 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (10793 ms) 2024-03-28T16:19:27.863 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-03-28T16:19:27.863 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (153 ms) 2024-03-28T16:19:27.863 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-03-28T16:19:27.863 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (80 ms) 2024-03-28T16:19:27.863 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-03-28T16:19:27.863 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.TruncTest (233 ms) 2024-03-28T16:19:27.863 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-03-28T16:19:27.864 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (95 ms) 2024-03-28T16:19:27.864 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-03-28T16:19:27.864 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (1772 ms) 2024-03-28T16:19:27.864 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-03-28T16:19:27.864 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.RmXattr (744 ms) 2024-03-28T16:19:27.864 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-03-28T16:19:27.864 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIo.XattrIter (181 ms) 2024-03-28T16:19:27.864 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [----------] 14 tests from LibRadosIo (87234 ms total) 2024-03-28T16:19:27.864 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: 2024-03-28T16:19:27.864 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-03-28T16:19:27.864 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-03-28T16:19:27.864 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (9024 ms) 2024-03-28T16:19:27.864 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-03-28T16:19:27.864 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (915 ms) 2024-03-28T16:19:27.864 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-03-28T16:19:27.864 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (540 ms) 2024-03-28T16:19:27.865 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-03-28T16:19:27.865 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (2298 ms) 2024-03-28T16:19:27.865 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-03-28T16:19:27.865 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (567 ms) 2024-03-28T16:19:27.865 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-03-28T16:19:27.865 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (294 ms) 2024-03-28T16:19:27.865 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-03-28T16:19:27.865 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (312 ms) 2024-03-28T16:19:27.865 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-03-28T16:19:27.865 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (160 ms) 2024-03-28T16:19:27.865 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-03-28T16:19:27.865 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (9844 ms) 2024-03-28T16:19:27.865 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-03-28T16:19:27.865 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (4768 ms) 2024-03-28T16:19:27.865 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [----------] 10 tests from LibRadosIoEC (28722 ms total) 2024-03-28T16:19:27.866 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: 2024-03-28T16:19:27.866 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [----------] Global test environment tear-down 2024-03-28T16:19:27.866 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [==========] 24 tests from 2 test suites ran. (160907 ms total) 2024-03-28T16:19:27.866 INFO:tasks.workunit.client.0.smithi007.stdout: api_io: [ PASSED ] 24 tests. 2024-03-28T16:19:36.645 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_1_[40847]: starting. 2024-03-28T16:19:36.645 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_1_[40847]: creating pool ceph_test_rados_list_parallel.smithi007-40819 2024-03-28T16:19:36.645 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_1_[40847]: created object 0... 2024-03-28T16:19:36.645 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_1_[40847]: created object 25... 2024-03-28T16:19:36.645 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_1_[40847]: created object 49... 2024-03-28T16:19:36.645 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_1_[40847]: finishing. 2024-03-28T16:19:36.645 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_1_[40847]: shutting down. 2024-03-28T16:19:36.645 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_2_[40848]: starting. 2024-03-28T16:19:36.645 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_2_[40848]: listing objects. 2024-03-28T16:19:36.646 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_2_[40848]: listed object 0... 2024-03-28T16:19:36.646 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_2_[40848]: listed object 25... 2024-03-28T16:19:36.646 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_2_[40848]: saw 50 objects 2024-03-28T16:19:36.646 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_2_[40848]: shutting down. 2024-03-28T16:19:36.646 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:19:36.646 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_3_[63286]: starting. 2024-03-28T16:19:36.646 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_3_[63286]: creating pool ceph_test_rados_list_parallel.smithi007-40819 2024-03-28T16:19:36.646 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_3_[63286]: created object 0... 2024-03-28T16:19:36.646 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_3_[63286]: created object 25... 2024-03-28T16:19:36.646 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_3_[63286]: created object 49... 2024-03-28T16:19:36.646 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_3_[63286]: finishing. 2024-03-28T16:19:36.646 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_3_[63286]: shutting down. 2024-03-28T16:19:36.646 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:19:36.647 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_4_[63287]: starting. 2024-03-28T16:19:36.647 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_4_[63287]: listing objects. 2024-03-28T16:19:36.647 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_4_[63287]: listed object 0... 2024-03-28T16:19:36.647 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_4_[63287]: listed object 25... 2024-03-28T16:19:36.647 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_4_[63287]: saw 45 objects 2024-03-28T16:19:36.647 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_4_[63287]: shutting down. 2024-03-28T16:19:36.647 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:19:36.647 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_5_[63288]: starting. 2024-03-28T16:19:36.647 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_5_[63288]: removed 25 objects... 2024-03-28T16:19:36.647 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_5_[63288]: removed half of the objects 2024-03-28T16:19:36.647 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_5_[63288]: removed 50 objects... 2024-03-28T16:19:36.647 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_5_[63288]: removed 50 objects 2024-03-28T16:19:36.647 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_5_[63288]: shutting down. 2024-03-28T16:19:36.647 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:19:36.647 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:19:36.647 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_6_[71964]: starting. 2024-03-28T16:19:36.648 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_6_[71964]: creating pool ceph_test_rados_list_parallel.smithi007-40819 2024-03-28T16:19:36.648 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_6_[71964]: created object 0... 2024-03-28T16:19:36.648 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_6_[71964]: created object 25... 2024-03-28T16:19:36.648 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_6_[71964]: created object 49... 2024-03-28T16:19:36.648 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_6_[71964]: finishing. 2024-03-28T16:19:36.648 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_6_[71964]: shutting down. 2024-03-28T16:19:36.648 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:19:36.648 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:19:36.648 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_7_[71966]: starting. 2024-03-28T16:19:36.648 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_7_[71966]: listing objects. 2024-03-28T16:19:36.648 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_7_[71966]: listed object 0... 2024-03-28T16:19:36.648 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_7_[71966]: listed object 25... 2024-03-28T16:19:36.648 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_7_[71966]: listed object 50... 2024-03-28T16:19:36.648 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_7_[71966]: saw 55 objects 2024-03-28T16:19:36.649 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_7_[71966]: shutting down. 2024-03-28T16:19:36.649 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:19:36.649 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:19:36.649 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_8_[71967]: starting. 2024-03-28T16:19:36.649 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_8_[71967]: added 25 objects... 2024-03-28T16:19:36.649 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_8_[71967]: added half of the objects 2024-03-28T16:19:36.649 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_8_[71967]: added 50 objects... 2024-03-28T16:19:36.649 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_8_[71967]: added 50 objects 2024-03-28T16:19:36.649 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_8_[71967]: shutting down. 2024-03-28T16:19:36.649 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:19:36.649 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:19:40.812 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ********************** api_watch_notify_pp: Running main() from gmock_main.cc 2024-03-28T16:19:40.813 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-03-28T16:19:40.813 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-03-28T16:19:40.813 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-03-28T16:19:40.813 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-03-28T16:19:40.813 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: notify 2024-03-28T16:19:40.813 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (44588 ms) 2024-03-28T16:19:40.813 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-03-28T16:19:40.813 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (30046 ms) 2024-03-28T16:19:40.813 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (74634 ms total) 2024-03-28T16:19:40.813 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: 2024-03-28T16:19:40.813 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-03-28T16:19:40.813 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-03-28T16:19:40.813 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: notify 2024-03-28T16:19:40.814 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (3055 ms) 2024-03-28T16:19:40.814 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-03-28T16:19:40.814 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: notify 2024-03-28T16:19:40.814 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (24559 ms) 2024-03-28T16:19:40.814 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-03-28T16:19:40.814 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (549 ms) 2024-03-28T16:19:40.814 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-03-28T16:19:40.814 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (3610 ms) 2024-03-28T16:19:40.814 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-03-28T16:19:40.814 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 93907888050704 notify_id 558345748480 notifier_gid 9654 2024-03-28T16:19:40.814 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (2827 ms) 2024-03-28T16:19:40.814 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-03-28T16:19:40.815 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 93907888049360 notify_id 562640715776 notifier_gid 9654 2024-03-28T16:19:40.815 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (1146 ms) 2024-03-28T16:19:40.815 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-03-28T16:19:40.815 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 93907888049360 notify_id 571230650369 notifier_gid 9654 2024-03-28T16:19:40.815 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (2450 ms) 2024-03-28T16:19:40.815 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-03-28T16:19:40.815 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 93907888049360 notify_id 575525617665 notifier_gid 9654 2024-03-28T16:19:40.815 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (386 ms) 2024-03-28T16:19:40.815 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-03-28T16:19:40.815 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 93907888634720 notify_id 584115552258 notifier_gid 9654 2024-03-28T16:19:40.815 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (1784 ms) 2024-03-28T16:19:40.815 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-03-28T16:19:40.815 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 93907888634720 notify_id 588410519554 notifier_gid 9654 2024-03-28T16:19:40.815 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (1173 ms) 2024-03-28T16:19:40.816 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-03-28T16:19:40.816 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: trying... 2024-03-28T16:19:40.816 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 93907888634720 notify_id 597000454150 notifier_gid 9654 2024-03-28T16:19:40.816 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: timed out 2024-03-28T16:19:49.741 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:22.314 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:19:49.743 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:22.316 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:19:49.806 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:05:22.363 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:19:49.810 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:05:22.366 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:19:49.825 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:22.398 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:19:49.833 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:22.406 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:19:49.926 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:05:22.482 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:19:49.981 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:22.553 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:19:50.000 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:22.572 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:19:55.901 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: flushing 2024-03-28T16:19:55.901 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: flushed 2024-03-28T16:19:55.901 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (5004 ms) 2024-03-28T16:19:55.901 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-03-28T16:19:55.901 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: trying... 2024-03-28T16:19:55.901 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 93907888634720 notify_id 614180323331 notifier_gid 9654 2024-03-28T16:19:55.901 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: timed out 2024-03-28T16:19:55.901 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: flushing 2024-03-28T16:19:55.901 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: flushed 2024-03-28T16:19:55.902 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3545 ms) 2024-03-28T16:19:55.902 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-03-28T16:19:55.902 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: List watches 2024-03-28T16:19:55.902 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: notify2 2024-03-28T16:19:55.902 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 93907888634720 notify_id 627065225220 notifier_gid 9654 2024-03-28T16:19:55.902 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: notify2 done 2024-03-28T16:19:55.902 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: watch_check 2024-03-28T16:19:55.902 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: unwatch2 2024-03-28T16:19:55.902 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: flushing 2024-03-28T16:19:55.902 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: done 2024-03-28T16:19:55.902 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3639 ms) 2024-03-28T16:19:55.902 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-03-28T16:19:55.902 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: List watches 2024-03-28T16:19:55.902 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: notify2 2024-03-28T16:19:55.903 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: handle_notify cookie 93907888634720 notify_id 639950127109 notifier_gid 9654 2024-03-28T16:19:55.903 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: notify2 done 2024-03-28T16:19:55.903 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: watch_check 2024-03-28T16:19:55.903 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: unwatch2 2024-03-28T16:19:55.903 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: flushing 2024-03-28T16:19:55.903 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: done 2024-03-28T16:19:55.903 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3225 ms) 2024-03-28T16:19:55.903 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (56953 ms total) 2024-03-28T16:19:55.903 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: 2024-03-28T16:19:55.903 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-03-28T16:19:55.903 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (188724 ms total) 2024-03-28T16:19:55.903 INFO:tasks.workunit.client.0.smithi007.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-03-28T16:20:14.826 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:47.398 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:14.879 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:05:47.435 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:14.882 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:05:47.438 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:14.889 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:47.462 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:14.938 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:05:47.494 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:14.951 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:47.524 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:14.953 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:47.526 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:15.009 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:47.582 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:15.028 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:47.601 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:15.284 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: Running main() from gmock_main.cc 2024-03-28T16:20:15.285 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-03-28T16:20:15.285 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] Global test environment set-up. 2024-03-28T16:20:15.285 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-03-28T16:20:15.285 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-03-28T16:20:15.285 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (27381 ms) 2024-03-28T16:20:15.285 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-03-28T16:20:15.285 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (13009 ms) 2024-03-28T16:20:15.285 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-03-28T16:20:15.285 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (5926 ms) 2024-03-28T16:20:15.285 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-03-28T16:20:15.285 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (5293 ms) 2024-03-28T16:20:15.285 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (51609 ms total) 2024-03-28T16:20:15.285 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: 2024-03-28T16:20:15.285 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-03-28T16:20:15.286 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-03-28T16:20:15.286 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (6905 ms) 2024-03-28T16:20:15.286 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-03-28T16:20:15.286 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (14643 ms) 2024-03-28T16:20:15.286 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-03-28T16:20:15.286 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (51664 ms) 2024-03-28T16:20:15.286 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (73212 ms total) 2024-03-28T16:20:15.286 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: 2024-03-28T16:20:15.286 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-03-28T16:20:15.286 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-03-28T16:20:15.286 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (10112 ms) 2024-03-28T16:20:15.286 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-03-28T16:20:15.286 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (3106 ms) 2024-03-28T16:20:15.286 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-03-28T16:20:15.287 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (4162 ms) 2024-03-28T16:20:15.287 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-03-28T16:20:15.287 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2806 ms) 2024-03-28T16:20:15.287 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (20186 ms total) 2024-03-28T16:20:15.287 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: 2024-03-28T16:20:15.287 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-03-28T16:20:15.287 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-03-28T16:20:15.287 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (8544 ms) 2024-03-28T16:20:15.287 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-03-28T16:20:15.287 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (20241 ms) 2024-03-28T16:20:15.287 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (28785 ms total) 2024-03-28T16:20:15.287 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: 2024-03-28T16:20:15.287 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [----------] Global test environment tear-down 2024-03-28T16:20:15.287 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (208188 ms total) 2024-03-28T16:20:15.288 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-03-28T16:20:18.169 INFO:tasks.workunit.client.0.smithi007.stdout:********* 2024-03-28T16:20:18.169 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_9_[102628]: starting. 2024-03-28T16:20:18.169 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_9_[102628]: creating pool ceph_test_rados_list_parallel.smithi007-40819 2024-03-28T16:20:18.169 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_9_[102628]: created object 0... 2024-03-28T16:20:18.169 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_9_[102628]: created object 25... 2024-03-28T16:20:18.169 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_9_[102628]: created object 49... 2024-03-28T16:20:18.170 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_9_[102628]: finishing. 2024-03-28T16:20:18.170 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_9_[102628]: shutting down. 2024-03-28T16:20:18.170 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:18.170 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:18.170 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:18.170 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[102629]: starting. 2024-03-28T16:20:18.170 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[102629]: listing objects. 2024-03-28T16:20:18.170 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[102629]: listed object 0... 2024-03-28T16:20:18.170 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[102629]: listed object 25... 2024-03-28T16:20:18.170 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[102629]: listed object 50... 2024-03-28T16:20:18.170 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[102629]: listed object 75... 2024-03-28T16:20:18.170 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[102629]: saw 97 objects 2024-03-28T16:20:18.170 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_10_[102629]: shutting down. 2024-03-28T16:20:18.171 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:18.171 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:18.171 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:18.171 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_11_[102630]: starting. 2024-03-28T16:20:18.171 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_11_[102630]: added 25 objects... 2024-03-28T16:20:18.171 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_11_[102630]: added half of the objects 2024-03-28T16:20:18.171 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_11_[102630]: added 50 objects... 2024-03-28T16:20:18.171 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_11_[102630]: added 50 objects 2024-03-28T16:20:18.171 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_11_[102630]: shutting down. 2024-03-28T16:20:18.171 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:18.171 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:18.171 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:18.171 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_13_[102632]: starting. 2024-03-28T16:20:18.171 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_13_[102632]: removed 25 objects... 2024-03-28T16:20:18.171 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_13_[102632]: removed half of the objects 2024-03-28T16:20:18.171 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_13_[102632]: removed 50 objects... 2024-03-28T16:20:18.172 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_13_[102632]: removed 50 objects 2024-03-28T16:20:18.172 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_13_[102632]: shutting down. 2024-03-28T16:20:18.172 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:18.172 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:18.172 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:18.172 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_12_[102631]: starting. 2024-03-28T16:20:18.172 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_12_[102631]: added 25 objects... 2024-03-28T16:20:18.172 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_12_[102631]: added half of the objects 2024-03-28T16:20:18.172 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_12_[102631]: added 50 objects... 2024-03-28T16:20:18.172 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_12_[102631]: added 50 objects 2024-03-28T16:20:18.172 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_12_[102631]: shutting down. 2024-03-28T16:20:18.172 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:18.172 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:18.172 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:18.172 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:18.172 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_14_[104534]: starting. 2024-03-28T16:20:18.173 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_14_[104534]: creating pool ceph_test_rados_list_parallel.smithi007-40819 2024-03-28T16:20:18.173 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_14_[104534]: created object 0... 2024-03-28T16:20:18.173 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_14_[104534]: created object 25... 2024-03-28T16:20:18.173 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_14_[104534]: created object 49... 2024-03-28T16:20:18.173 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_14_[104534]: finishing. 2024-03-28T16:20:18.173 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_14_[104534]: shutting down. 2024-03-28T16:20:18.173 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:18.173 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:18.173 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:18.173 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:18.173 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[104535]: starting. 2024-03-28T16:20:18.173 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[104535]: listing objects. 2024-03-28T16:20:18.173 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[104535]: listed object 0... 2024-03-28T16:20:35.286 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:07.859 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:35.294 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:07.867 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:35.296 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:07.869 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:35.355 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:07.928 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:35.365 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:07.938 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:35.422 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:07.995 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:35.423 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:06:07.980 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:35.425 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:06:07.981 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:35.504 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:06:08.060 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:52.546 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:20:52.446+0000 12260640 -1 bdev(0x1519e530 /var/lib/ceph/osd/ceph-4/block) debug_aio_unlink stalled aio 0x65a8080 since 2024-03-28T16:20:36.247756+0000, timeout is 5s 2024-03-28T16:20:53.189 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[104535]: listed object 25... 2024-03-28T16:20:53.189 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[104535]: listed object 50... 2024-03-28T16:20:53.189 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[104535]: listed object 75... 2024-03-28T16:20:53.189 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[104535]: listed object 100... 2024-03-28T16:20:53.189 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[104535]: listed object 125... 2024-03-28T16:20:53.189 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[104535]: saw 150 objects 2024-03-28T16:20:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_15_[104535]: shutting down. 2024-03-28T16:20:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_16_[104536]: starting. 2024-03-28T16:20:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_16_[104536]: added 25 objects... 2024-03-28T16:20:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_16_[104536]: added half of the objects 2024-03-28T16:20:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_16_[104536]: added 50 objects... 2024-03-28T16:20:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_16_[104536]: added 50 objects 2024-03-28T16:20:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: process_16_[104536]: shutting down. 2024-03-28T16:20:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:53.190 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:53.191 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******************************* 2024-03-28T16:20:53.191 INFO:tasks.workunit.client.0.smithi007.stdout: list_parallel: ******* SUCCESS ********** 2024-03-28T16:20:55.091 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:27.664 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:55.096 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:27.669 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:55.143 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:06:27.699 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:55.146 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:27.719 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:55.147 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:27.720 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:55.147 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:06:27.703 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:55.148 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:27.721 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:55.201 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:27.774 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:55.211 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:06:27.767 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:20:55.703 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: Running main() from gmock_main.cc 2024-03-28T16:20:55.703 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-03-28T16:20:55.704 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [----------] Global test environment set-up. 2024-03-28T16:20:55.704 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-03-28T16:20:55.704 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-03-28T16:20:55.704 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.TooBig (5822 ms) 2024-03-28T16:20:55.704 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-03-28T16:20:55.704 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (24807 ms) 2024-03-28T16:20:55.704 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-03-28T16:20:55.704 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (20415 ms) 2024-03-28T16:20:55.704 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-03-28T16:20:55.704 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (14671 ms) 2024-03-28T16:20:55.704 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-03-28T16:20:55.704 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (5661 ms) 2024-03-28T16:20:55.704 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-03-28T16:20:55.705 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (29160 ms) 2024-03-28T16:20:55.705 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-03-28T16:20:55.705 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (37336 ms) 2024-03-28T16:20:55.705 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-03-28T16:20:55.705 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (11905 ms) 2024-03-28T16:20:55.705 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-03-28T16:20:55.705 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3397 ms) 2024-03-28T16:20:55.705 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-03-28T16:20:55.705 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (4148 ms) 2024-03-28T16:20:55.705 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-03-28T16:20:55.705 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (5627 ms) 2024-03-28T16:20:55.705 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-03-28T16:20:55.705 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (4175 ms) 2024-03-28T16:20:55.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-03-28T16:20:55.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3927 ms) 2024-03-28T16:20:55.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-03-28T16:20:55.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2752 ms) 2024-03-28T16:20:55.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-03-28T16:20:55.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.Flush (3253 ms) 2024-03-28T16:20:55.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-03-28T16:20:55.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (4168 ms) 2024-03-28T16:20:55.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-03-28T16:20:55.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (4412 ms) 2024-03-28T16:20:55.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-03-28T16:20:55.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (4349 ms) 2024-03-28T16:20:55.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-03-28T16:20:55.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (2916 ms) 2024-03-28T16:20:55.706 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-03-28T16:20:55.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3690 ms) 2024-03-28T16:20:55.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-03-28T16:20:55.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (4081 ms) 2024-03-28T16:20:55.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-03-28T16:20:55.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3136 ms) 2024-03-28T16:20:55.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-03-28T16:20:55.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (4492 ms) 2024-03-28T16:20:55.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-03-28T16:20:55.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3218 ms) 2024-03-28T16:20:55.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-03-28T16:20:55.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (2515 ms) 2024-03-28T16:20:55.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-03-28T16:20:55.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (34723 ms) 2024-03-28T16:20:55.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [----------] 26 tests from LibRadosAio (248756 ms total) 2024-03-28T16:20:55.707 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: 2024-03-28T16:20:55.708 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-03-28T16:21:00.248 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.Simpl api_snapshots_pp: Running main() from gmock_main.cc 2024-03-28T16:21:00.248 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-03-28T16:21:00.248 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-03-28T16:21:00.248 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-03-28T16:21:00.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: seed 40509 2024-03-28T16:21:00.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-03-28T16:21:00.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (87845 ms) 2024-03-28T16:21:00.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-03-28T16:21:00.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (15578 ms) 2024-03-28T16:21:00.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-03-28T16:21:00.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (20460 ms) 2024-03-28T16:21:00.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-03-28T16:21:00.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (16384 ms) 2024-03-28T16:21:00.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-03-28T16:21:00.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (12680 ms) 2024-03-28T16:21:00.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (152947 ms total) 2024-03-28T16:21:00.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: 2024-03-28T16:21:00.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-03-28T16:21:00.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-03-28T16:21:00.249 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4838 ms) 2024-03-28T16:21:00.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-03-28T16:21:00.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (6369 ms) 2024-03-28T16:21:00.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-03-28T16:21:00.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (8977 ms) 2024-03-28T16:21:00.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-03-28T16:21:00.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (5616 ms) 2024-03-28T16:21:00.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-03-28T16:21:00.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (15781 ms) 2024-03-28T16:21:00.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-28T16:21:00.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2-1053-ge58b9979/rpm/el9/BUILD/ceph-18.2.2-1053-ge58b9979/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-03-28T16:21:00.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: 2024-03-28T16:21:00.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-03-28T16:21:00.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-03-28T16:21:00.250 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi007-40509-7 2024-03-28T16:21:00.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: waiting for snaps to purge 2024-03-28T16:21:00.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (18520 ms) 2024-03-28T16:21:00.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (60101 ms total) 2024-03-28T16:21:00.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: 2024-03-28T16:21:00.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-03-28T16:21:00.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-03-28T16:21:00.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (5 ms) 2024-03-28T16:21:00.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-03-28T16:21:00.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5483 ms) 2024-03-28T16:21:00.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5488 ms total) 2024-03-28T16:21:00.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: 2024-03-28T16:21:00.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-03-28T16:21:00.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-03-28T16:21:00.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (19477 ms) 2024-03-28T16:21:00.251 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-03-28T16:21:20.148 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:52.720 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:21:20.152 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:52.725 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:21:20.184 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:52.757 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:21:20.197 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:06:52.754 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:21:20.200 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:06:52.756 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:21:20.204 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:52.777 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:21:20.206 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:52.779 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:21:20.229 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:06:52.785 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:21:20.235 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:52.808 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:21:21.964 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:21.961+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:22.973 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:22.970+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:23.962 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:23.959+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:24.924 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:24.922+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:25.902 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:25.900+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:26.884 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:26.882+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:27.737 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:21:27.735+0000 ea59640 -1 osd.4 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17101.0:302 103.7 103:e099f9bc:test-rados-api-smithi007-40356-12::bar:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e213) 2024-03-28T16:21:27.838 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:27.837+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:28.693 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:21:28.691+0000 ea59640 -1 osd.4 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.17101.0:302 103.7 103:e099f9bc:test-rados-api-smithi007-40356-12::bar:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e213) 2024-03-28T16:21:28.794 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:28.792+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:29.705 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:21:29.704+0000 ea59640 -1 osd.4 218 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.17101.0:302 103.7 103:e099f9bc:test-rados-api-smithi007-40356-12::bar:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e213) 2024-03-28T16:21:29.775 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:29.771+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:30.742 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:21:30.740+0000 ea59640 -1 osd.4 218 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.17101.0:302 103.7 103:e099f9bc:test-rados-api-smithi007-40356-12::bar:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e213) 2024-03-28T16:21:30.753 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:30.753+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:31.695 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:21:31.694+0000 ea59640 -1 osd.4 218 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.17101.0:302 103.7 103:e099f9bc:test-rados-api-smithi007-40356-12::bar:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e213) 2024-03-28T16:21:31.752 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:31.751+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:32.693 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:21:32.693+0000 ea59640 -1 osd.4 218 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.17101.0:302 103.7 103:e099f9bc:test-rados-api-smithi007-40356-12::bar:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e213) 2024-03-28T16:21:32.775 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:32.774+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:33.663 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:21:33.662+0000 ea59640 -1 osd.4 218 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.17101.0:302 103.7 103:e099f9bc:test-rados-api-smithi007-40356-12::bar:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e213) 2024-03-28T16:21:33.799 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:33.799+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:34.689 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:21:34.688+0000 ea59640 -1 osd.4 218 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.17101.0:302 103.7 103:e099f9bc:test-rados-api-smithi007-40356-12::bar:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e213) 2024-03-28T16:21:34.830 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:34.828+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:35.667 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:21:35.666+0000 ea59640 -1 osd.4 218 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.17101.0:302 103.7 103:e099f9bc:test-rados-api-smithi007-40356-12::bar:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e213) 2024-03-28T16:21:35.790 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:35.789+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:36.633 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:21:36.632+0000 ea59640 -1 osd.4 218 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.17101.0:302 103.7 103:e099f9bc:test-rados-api-smithi007-40356-12::bar:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e213) 2024-03-28T16:21:36.752 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:36.751+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:37.629 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:21:37.629+0000 ea59640 -1 osd.4 218 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.17101.0:302 103.7 103:e099f9bc:test-rados-api-smithi007-40356-12::bar:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e213) 2024-03-28T16:21:37.749 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:37.747+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:38.659 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:21:38.658+0000 ea59640 -1 osd.4 218 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.17101.0:302 103.7 103:e099f9bc:test-rados-api-smithi007-40356-12::bar:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e213) 2024-03-28T16:21:38.760 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:38.759+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:39.689 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:21:39.688+0000 ea59640 -1 osd.4 218 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.17101.0:302 103.7 103:e099f9bc:test-rados-api-smithi007-40356-12::bar:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e213) 2024-03-28T16:21:39.721 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:39.720+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:40.253 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:12.826 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:21:40.257 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:12.830 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:21:40.283 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:12.856 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:21:40.304 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:07:12.861 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:21:40.306 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:07:12.863 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:21:40.309 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:12.882 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:21:40.313 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:12.886 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:21:40.329 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:07:12.886 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:21:40.339 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:12.912 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:21:40.656 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:21:40.655+0000 ea59640 -1 osd.4 218 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.17101.0:302 103.7 103:e099f9bc:test-rados-api-smithi007-40356-12::bar:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e213) 2024-03-28T16:21:40.764 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:40.763+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:41.710 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:21:41.709+0000 ea59640 -1 osd.4 218 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.17101.0:302 103.7 103:e099f9bc:test-rados-api-smithi007-40356-12::bar:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e213) 2024-03-28T16:21:41.741 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:41.741+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:42.311 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:21:42.302+0000 ea59640 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.83:6830 osd.5 since back 2024-03-28T16:20:12.955607+0000 front 2024-03-28T16:20:12.961341+0000 (oldest deadline 2024-03-28T16:21:38.602224+0000) 2024-03-28T16:21:42.674 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:21:42.673+0000 ea59640 -1 osd.4 218 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.17101.0:302 103.7 103:e099f9bc:test-rados-api-smithi007-40356-12::bar:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e213) 2024-03-28T16:21:42.774 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:42.774+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:43.402 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:21:43.366+0000 ea59640 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.83:6830 osd.5 since back 2024-03-28T16:21:43.048766+0000 front 2024-03-28T16:20:12.961341+0000 (oldest deadline 2024-03-28T16:21:38.602224+0000) 2024-03-28T16:21:43.657 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:21:43.655+0000 ea59640 -1 osd.4 218 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.17101.0:302 103.7 103:e099f9bc:test-rados-api-smithi007-40356-12::bar:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e213) 2024-03-28T16:21:43.754 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:43.751+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:44.468 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:21:44.391+0000 ea59640 -1 osd.7 218 heartbeat_check: no reply from 172.21.15.83:6830 osd.5 since back 2024-03-28T16:21:43.048766+0000 front 2024-03-28T16:20:12.961341+0000 (oldest deadline 2024-03-28T16:21:38.602224+0000) 2024-03-28T16:21:44.654 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:21:44.653+0000 ea59640 -1 osd.4 218 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.17101.0:302 103.7 103:e099f9bc:test-rados-api-smithi007-40356-12::bar:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e213) 2024-03-28T16:21:44.746 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:44.744+0000 ea59640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:45.608 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:21:45.605+0000 ea59640 -1 osd.7 219 heartbeat_check: no reply from 172.21.15.83:6830 osd.5 since back 2024-03-28T16:21:43.048766+0000 front 2024-03-28T16:21:45.223858+0000 (oldest deadline 2024-03-28T16:21:38.602224+0000) 2024-03-28T16:21:45.743 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:21:45.742+0000 ea59640 -1 osd.3 220 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21058.0:2 124.1a 124.327daa3a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e209) 2024-03-28T16:21:46.402 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] L api_aio_pp: Running main() from gmock_main.cc 2024-03-28T16:21:46.402 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-03-28T16:21:46.402 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-03-28T16:21:46.402 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-03-28T16:21:46.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-03-28T16:21:46.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (4341 ms) 2024-03-28T16:21:46.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-03-28T16:21:46.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (42512 ms) 2024-03-28T16:21:46.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-03-28T16:21:46.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (15342 ms) 2024-03-28T16:21:46.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-03-28T16:21:46.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (4652 ms) 2024-03-28T16:21:46.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-03-28T16:21:46.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (27959 ms) 2024-03-28T16:21:46.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-03-28T16:21:46.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (33534 ms) 2024-03-28T16:21:46.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-03-28T16:21:46.403 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (22462 ms) 2024-03-28T16:21:46.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-03-28T16:21:46.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (5477 ms) 2024-03-28T16:21:46.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-03-28T16:21:46.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (5611 ms) 2024-03-28T16:21:46.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-03-28T16:21:46.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3134 ms) 2024-03-28T16:21:46.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-03-28T16:21:46.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3657 ms) 2024-03-28T16:21:46.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-03-28T16:21:46.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (4081 ms) 2024-03-28T16:21:46.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-03-28T16:21:46.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3311 ms) 2024-03-28T16:21:46.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-03-28T16:21:46.404 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3946 ms) 2024-03-28T16:21:46.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-03-28T16:21:46.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3735 ms) 2024-03-28T16:21:46.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-03-28T16:21:46.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3982 ms) 2024-03-28T16:21:46.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-03-28T16:21:46.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (16137 ms) 2024-03-28T16:21:46.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-03-28T16:21:46.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3230 ms) 2024-03-28T16:21:46.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-03-28T16:21:46.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3365 ms) 2024-03-28T16:21:46.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-03-28T16:21:46.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3557 ms) 2024-03-28T16:21:46.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-03-28T16:21:46.405 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (11085 ms) 2024-03-28T16:21:46.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-03-28T16:21:46.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (4113 ms) 2024-03-28T16:21:46.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-03-28T16:21:46.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3776 ms) 2024-03-28T16:21:46.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-03-28T16:21:46.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3440 ms) 2024-03-28T16:21:46.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-03-28T16:21:46.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3158 ms) 2024-03-28T16:21:46.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-03-28T16:21:46.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3249 ms) 2024-03-28T16:21:46.406 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-03-28T16:22:00.509 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:33.081 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:00.510 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:33.083 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:00.559 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:07:33.115 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:00.560 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:07:33.117 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:00.564 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:33.136 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:00.566 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:33.139 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:00.568 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:33.141 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:00.605 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:07:33.162 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:00.621 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:33.194 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:20.451 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:53.023 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:20.457 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:53.030 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:20.494 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:53.066 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:20.501 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:07:53.057 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:20.502 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:07:53.058 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:20.516 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:53.089 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:20.521 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:53.094 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:20.526 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:07:53.082 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:20.544 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:53.116 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:22.277 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:22:22.272+0000 ea59640 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21070.0:1 129.6s0 129.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e226) 2024-03-28T16:22:23.282 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:22:23.282+0000 ea59640 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21070.0:1 129.6s0 129.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e226) 2024-03-28T16:22:24.322 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:22:24.321+0000 ea59640 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21070.0:1 129.6s0 129.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e226) 2024-03-28T16:22:25.302 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:22:25.301+0000 ea59640 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21070.0:1 129.6s0 129.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e226) 2024-03-28T16:22:26.322 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:22:26.321+0000 ea59640 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21070.0:1 129.6s0 129.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e226) 2024-03-28T16:22:27.287 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:22:27.287+0000 ea59640 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21070.0:1 129.6s0 129.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e226) 2024-03-28T16:22:27.861 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:27.861+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:22:28.282 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:22:28.281+0000 ea59640 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21070.0:1 129.6s0 129.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e226) 2024-03-28T16:22:28.831 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:28.831+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:22:29.383 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:22:29.381+0000 ea59640 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21070.0:1 129.6s0 129.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e226) 2024-03-28T16:22:29.850 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:29.850+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:22:30.861 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:30.861+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:22:31.840 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:31.839+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:22:32.824 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:32.822+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:22:33.834 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:33.832+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:22:34.876 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:34.873+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:22:35.895 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:35.893+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:22:36.902 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:36.901+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:22:37.304 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:09.877 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:37.307 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:09.880 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:37.335 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:09.908 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:37.362 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:08:09.919 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:37.365 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:09.938 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:37.366 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:09.939 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:37.368 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:08:09.924 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:37.401 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:08:09.958 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:37.405 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:09.979 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:37.889 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:37.888+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:22:38.872 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:38.871+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:22:39.842 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:39.839+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:22:40.534 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:13.106 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:40.539 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:13.112 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:40.550 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:13.123 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:40.582 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:08:13.138 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:40.583 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:08:13.140 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:40.589 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:13.162 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:40.590 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:13.163 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:40.593 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:08:13.150 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:40.601 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:13.174 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:22:40.860 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:40.859+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:22:41.869 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:41.866+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:22:42.912 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:42.910+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:22:43.953 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:43.952+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:22:44.931 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:44.930+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:22:45.958 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:45.957+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:22:47.008 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:47.006+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:22:48.032 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:48.029+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:22:48.660 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:22:48.659+0000 ea59640 -1 osd.7 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21070.0:1 129.6s0 129.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e226) 2024-03-28T16:22:49.038 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:22:49.037+0000 ea59640 -1 osd.4 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21085.0:2 130.5s0 130.e5a2ae3d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e232) 2024-03-28T16:23:00.785 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:33.358 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:00.789 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:33.362 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:00.834 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:08:33.390 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:00.836 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:08:33.392 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:00.852 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:33.425 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:00.854 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:33.427 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:00.857 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:33.430 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:00.877 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:08:33.434 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:00.917 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:33.490 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:07.853 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRaibRadosSnapshotsECPP.SnapRemovePP (4547 ms) 2024-03-28T16:23:07.854 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-03-28T16:23:07.854 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (48219 ms) 2024-03-28T16:23:07.854 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-03-28T16:23:07.854 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (1954 ms) 2024-03-28T16:23:07.854 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (74197 ms total) 2024-03-28T16:23:07.854 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: 2024-03-28T16:23:07.854 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-03-28T16:23:07.854 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-03-28T16:23:07.854 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (56477 ms) 2024-03-28T16:23:07.854 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-03-28T16:23:07.854 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (5036 ms) 2024-03-28T16:23:07.854 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-03-28T16:23:07.854 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (7186 ms) 2024-03-28T16:23:07.854 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (68699 ms total) 2024-03-28T16:23:07.855 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: 2024-03-28T16:23:07.855 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-03-28T16:23:07.855 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (380750 ms total) 2024-03-28T16:23:07.855 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-03-28T16:23:07.855 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-03-28T16:23:07.855 INFO:tasks.workunit.client.0.smithi007.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-28T16:23:20.937 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:53.509 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:20.940 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:53.513 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:21.004 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:53.577 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:21.006 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:53.579 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:21.011 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:53.584 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:21.014 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:08:53.571 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:21.023 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:08:53.579 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:21.083 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:53.655 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:21.092 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:08:53.649 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:29.017 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-03-28T16:23:29.017 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-03-28T16:23:29.017 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-03-28T16:23:29.017 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-03-28T16:23:29.017 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-03-28T16:23:29.017 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-28T16:23:29.017 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: 2024-03-28T16:23:29.017 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-03-28T16:23:29.017 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: seed 40313 2024-03-28T16:23:29.017 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-03-28T16:23:29.017 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (12 ms) 2024-03-28T16:23:29.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-03-28T16:23:29.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (32668 ms) 2024-03-28T16:23:29.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-03-28T16:23:29.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (24618 ms) 2024-03-28T16:23:29.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-03-28T16:23:29.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (2511 ms) 2024-03-28T16:23:29.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-03-28T16:23:29.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (16766 ms) 2024-03-28T16:23:29.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-03-28T16:23:29.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (5639 ms) 2024-03-28T16:23:29.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-03-28T16:23:29.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (328 ms) 2024-03-28T16:23:29.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-03-28T16:23:29.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (146 ms) 2024-03-28T16:23:29.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-03-28T16:23:29.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (186 ms) 2024-03-28T16:23:29.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-03-28T16:23:29.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (218 ms) 2024-03-28T16:23:29.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-03-28T16:23:29.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (59 ms) 2024-03-28T16:23:29.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-03-28T16:23:29.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (179 ms) 2024-03-28T16:23:29.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-03-28T16:23:29.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (226 ms) 2024-03-28T16:23:29.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-03-28T16:23:29.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: osd_max_attr_size = 0 2024-03-28T16:23:29.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-03-28T16:23:29.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (131115 ms) 2024-03-28T16:23:29.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-03-28T16:23:29.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (87244 ms) 2024-03-28T16:23:29.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-03-28T16:23:29.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-03-28T16:23:29.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: done waiting, doing copies 2024-03-28T16:23:29.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: waiting for final deep scrubs... 2024-03-28T16:23:29.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: done waiting 2024-03-28T16:23:29.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (89610 ms) 2024-03-28T16:23:29.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-03-28T16:23:29.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (95 ms) 2024-03-28T16:23:29.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-03-28T16:23:29.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (40 ms) 2024-03-28T16:23:29.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-03-28T16:23:29.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (5250 ms) 2024-03-28T16:23:29.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-03-28T16:23:29.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-03-28T16:23:29.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-03-28T16:23:29.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-03-28T16:23:29.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-03-28T16:23:29.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-03-28T16:23:40.945 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:13.518 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:40.947 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:13.520 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:41.001 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:13.574 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:41.002 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:09:13.558 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:41.003 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:09:13.560 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:41.007 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:13.580 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:41.012 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:13.585 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:41.061 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:09:13.617 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:23:41.068 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:13.641 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:01.061 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:33.633 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:01.063 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:33.636 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:01.107 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:09:33.664 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:01.109 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:09:33.665 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:01.124 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:33.697 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:01.135 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:33.708 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:01.137 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:33.710 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:01.145 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:09:33.701 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:01.174 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:33.747 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:17.185 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:17.182+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:18.234 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:18.231+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:19.213 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:19.211+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:20.232 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:20.230+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:21.104 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:53.677 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:21.110 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:53.683 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:21.130 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:53.702 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:21.154 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:09:53.710 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:21.155 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:09:53.711 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:21.165 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:53.738 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:21.166 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:53.739 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:21.173 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:09:53.730 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:21.185 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:53.758 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:21.198 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:21.196+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:22.171 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:22.169+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:23.134 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:23.132+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:24.173 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:24.171+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:25.219 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:25.217+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:26.188 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:26.185+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:27.183 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:27.181+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:28.145 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:28.144+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:29.193 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:29.190+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:30.173 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:30.172+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:31.176 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:31.174+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:32.137 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:32.135+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:33.101 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:33.099+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:34.070 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:34.069+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:35.114 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:35.111+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:36.158 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:36.156+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:37.138 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:37.136+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:38.138 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:38.136+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:39.143 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:39.140+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:40.115 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:40.113+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:41.121 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:41.119+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:41.154 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:13.727 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:41.159 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:13.731 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:41.189 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:13.762 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:41.205 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:10:13.761 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:41.206 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:10:13.763 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:41.217 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:13.790 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:41.220 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:13.793 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:41.226 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:10:13.782 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:41.240 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:13.813 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:42.080 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:42.080+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:43.103 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:43.101+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:44.117 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:44.116+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:45.129 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:45.129+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:46.158 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:46.156+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:47.188 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:47.187+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:48.183 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:48.183+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:49.229 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:49.229+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:50.213 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:50.212+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:51.208 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:51.207+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:52.262 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:52.260+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:53.244 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:53.242+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:54.262 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:54.262+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:55.294 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:55.294+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:55.335 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:27.908 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:55.338 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:27.911 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:55.343 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:10:27.899 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:55.346 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:10:27.903 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:55.362 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:27.935 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:55.368 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:10:27.924 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:55.394 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:27.967 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:55.395 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:27.968 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:55.420 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:27.993 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:24:56.296 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:56.294+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:57.336 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:57.335+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:58.297 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:58.296+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:59.310 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:24:59.308+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:24:59.996 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-03-28T16:24:59.985+0000 ea59640 -1 osd.1 298 heartbeat_check: no reply from 172.21.15.83:6814 osd.6 since back 2024-03-28T16:23:44.661888+0000 front 2024-03-28T16:23:37.847786+0000 (oldest deadline 2024-03-28T16:24:59.934909+0000) 2024-03-28T16:25:00.365 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:25:00.363+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:25:00.718 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:25:00.713+0000 12a61640 -1 osd.5 298 heartbeat_check: no reply from 172.21.15.83:6814 osd.6 since back 2024-03-28T16:23:37.700378+0000 front 2024-03-28T16:23:37.703777+0000 (oldest deadline 2024-03-28T16:25:00.614930+0000) 2024-03-28T16:25:01.031 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-03-28T16:25:01.029+0000 ea59640 -1 osd.1 298 heartbeat_check: no reply from 172.21.15.83:6814 osd.6 since back 2024-03-28T16:23:44.661888+0000 front 2024-03-28T16:23:37.847786+0000 (oldest deadline 2024-03-28T16:24:59.934909+0000) 2024-03-28T16:25:01.170 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:25:01.166+0000 ea59640 -1 osd.2 298 heartbeat_check: no reply from 172.21.15.83:6814 osd.6 since back 2024-03-28T16:23:36.257575+0000 front 2024-03-28T16:23:36.244328+0000 (oldest deadline 2024-03-28T16:25:00.862934+0000) 2024-03-28T16:25:01.202 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:33.775 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:01.206 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:33.779 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:01.218 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:33.791 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:01.254 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:10:33.810 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:01.254 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:33.827 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:01.255 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:33.829 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:01.259 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:10:33.816 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:01.270 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:10:33.827 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:01.272 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:33.845 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:01.355 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:25:01.348+0000 ea59640 -1 osd.3 298 heartbeat_check: no reply from 172.21.15.83:6814 osd.6 since back 2024-03-28T16:23:37.537394+0000 front 2024-03-28T16:23:37.407188+0000 (oldest deadline 2024-03-28T16:25:00.764110+0000) 2024-03-28T16:25:01.361 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:25:01.358+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:25:01.759 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:25:01.756+0000 12a61640 -1 osd.5 298 heartbeat_check: no reply from 172.21.15.83:6814 osd.6 since back 2024-03-28T16:23:37.700378+0000 front 2024-03-28T16:23:37.703777+0000 (oldest deadline 2024-03-28T16:25:00.614930+0000) 2024-03-28T16:25:02.068 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-03-28T16:25:02.067+0000 ea59640 -1 osd.1 298 heartbeat_check: no reply from 172.21.15.83:6814 osd.6 since back 2024-03-28T16:23:44.661888+0000 front 2024-03-28T16:23:37.847786+0000 (oldest deadline 2024-03-28T16:24:59.934909+0000) 2024-03-28T16:25:02.174 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:25:02.172+0000 ea59640 -1 osd.2 298 heartbeat_check: no reply from 172.21.15.83:6814 osd.6 since back 2024-03-28T16:23:36.257575+0000 front 2024-03-28T16:23:36.244328+0000 (oldest deadline 2024-03-28T16:25:00.862934+0000) 2024-03-28T16:25:02.381 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:25:02.379+0000 ea59640 -1 osd.3 298 heartbeat_check: no reply from 172.21.15.83:6814 osd.6 since back 2024-03-28T16:23:37.537394+0000 front 2024-03-28T16:23:37.407188+0000 (oldest deadline 2024-03-28T16:25:00.764110+0000) 2024-03-28T16:25:02.381 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:25:02.380+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:25:02.812 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:25:02.810+0000 12a61640 -1 osd.5 298 heartbeat_check: no reply from 172.21.15.83:6814 osd.6 since back 2024-03-28T16:23:37.700378+0000 front 2024-03-28T16:23:37.703777+0000 (oldest deadline 2024-03-28T16:25:00.614930+0000) 2024-03-28T16:25:02.907 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-03-28T16:25:02.900+0000 ea59640 -1 osd.0 298 heartbeat_check: no reply from 172.21.15.83:6814 osd.6 since back 2024-03-28T16:23:36.093094+0000 front 2024-03-28T16:23:36.097508+0000 (oldest deadline 2024-03-28T16:25:01.990097+0000) 2024-03-28T16:25:03.004 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:25:03.002+0000 ea59640 -1 osd.7 298 heartbeat_check: no reply from 172.21.15.83:6814 osd.6 since back 2024-03-28T16:23:37.160804+0000 front 2024-03-28T16:23:37.177246+0000 (oldest deadline 2024-03-28T16:25:02.493962+0000) 2024-03-28T16:25:03.098 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-03-28T16:25:03.098+0000 ea59640 -1 osd.1 298 heartbeat_check: no reply from 172.21.15.83:6814 osd.6 since back 2024-03-28T16:23:44.661888+0000 front 2024-03-28T16:23:37.847786+0000 (oldest deadline 2024-03-28T16:24:59.934909+0000) 2024-03-28T16:25:03.180 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:25:03.174+0000 ea59640 -1 osd.2 298 heartbeat_check: no reply from 172.21.15.83:6814 osd.6 since back 2024-03-28T16:23:36.257575+0000 front 2024-03-28T16:23:36.244328+0000 (oldest deadline 2024-03-28T16:25:00.862934+0000) 2024-03-28T16:25:03.401 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:25:03.400+0000 ea59640 -1 osd.3 298 heartbeat_check: no reply from 172.21.15.83:6814 osd.6 since back 2024-03-28T16:23:37.537394+0000 front 2024-03-28T16:23:37.407188+0000 (oldest deadline 2024-03-28T16:25:00.764110+0000) 2024-03-28T16:25:03.401 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:25:03.400+0000 ea59640 -1 osd.3 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:25:03.794 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:25:03.793+0000 12a61640 -1 osd.5 298 heartbeat_check: no reply from 172.21.15.83:6814 osd.6 since back 2024-03-28T16:23:37.700378+0000 front 2024-03-28T16:23:37.703777+0000 (oldest deadline 2024-03-28T16:25:00.614930+0000) 2024-03-28T16:25:03.942 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-03-28T16:25:03.941+0000 ea59640 -1 osd.0 298 heartbeat_check: no reply from 172.21.15.83:6814 osd.6 since back 2024-03-28T16:23:36.093094+0000 front 2024-03-28T16:23:36.097508+0000 (oldest deadline 2024-03-28T16:25:01.990097+0000) 2024-03-28T16:25:03.988 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:25:03.987+0000 ea59640 -1 osd.7 298 heartbeat_check: no reply from 172.21.15.83:6814 osd.6 since back 2024-03-28T16:23:37.160804+0000 front 2024-03-28T16:23:37.177246+0000 (oldest deadline 2024-03-28T16:25:02.493962+0000) 2024-03-28T16:25:04.471 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:25:04.390+0000 ea59640 -1 osd.3 299 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:25:05.447 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:25:05.441+0000 ea59640 -1 osd.3 300 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.21199.0:1 154.6s0 154:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290) 2024-03-28T16:25:21.545 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:54.117 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:21.552 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:54.125 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:21.602 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:54.174 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:21.603 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:54.176 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:21.616 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:54.189 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:21.627 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:10:54.183 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:21.635 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:10:54.192 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:21.669 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:54.242 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:21.683 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:10:54.239 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:41.477 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:14.049 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:41.480 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:14.053 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:41.517 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:14.089 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:41.528 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:11:14.084 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:41.529 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:11:14.086 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:41.532 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:14.105 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:41.534 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:14.107 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:41.562 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:11:14.119 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:41.568 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:14.141 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:25:44.116 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:25:44.090+0000 15266640 -1 bdev(0x18ecbd00 /var/lib/ceph/osd/ceph-5/block) debug_aio_unlink stalled aio 0x1647d4a0 since 2024-03-28T16:25:24.035919+0000, timeout is 5s 2024-03-28T16:25:54.661 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP (396910 ms total) 2024-03-28T16:25:54.662 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: 2024-03-28T16:25:54.662 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-03-28T16:25:54.662 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-03-28T16:25:54.662 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (90305 ms) 2024-03-28T16:25:54.662 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (90305 ms total) 2024-03-28T16:25:54.662 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: 2024-03-28T16:25:54.662 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-03-28T16:25:54.662 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-03-28T16:25:54.662 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (306 ms) 2024-03-28T16:25:54.662 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-03-28T16:25:54.662 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (289 ms) 2024-03-28T16:25:54.662 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (595 ms total) 2024-03-28T16:25:54.662 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: 2024-03-28T16:25:54.663 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-03-28T16:25:54.663 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-03-28T16:25:54.663 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (330 ms) 2024-03-28T16:25:54.663 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-03-28T16:25:54.663 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (59 ms) 2024-03-28T16:25:54.663 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (389 ms total) 2024-03-28T16:25:54.663 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: 2024-03-28T16:25:54.663 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-03-28T16:25:54.663 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-03-28T16:25:54.663 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (429 ms) 2024-03-28T16:25:54.663 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-03-28T16:25:54.663 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (28665 ms) 2024-03-28T16:25:54.663 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (29094 ms total) 2024-03-28T16:25:54.663 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: 2024-03-28T16:25:54.663 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-03-28T16:25:54.663 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-03-28T16:25:54.664 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1458 ms) 2024-03-28T16:25:54.664 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1458 ms total) 2024-03-28T16:25:54.664 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: 2024-03-28T16:25:54.664 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-03-28T16:25:54.664 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (547624 ms total) 2024-03-28T16:25:54.664 INFO:tasks.workunit.client.0.smithi007.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-03-28T16:26:01.703 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:34.276 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:01.711 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:34.284 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:01.760 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:11:34.317 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:01.761 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:34.334 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:01.765 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:11:34.321 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:01.768 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:34.341 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:01.774 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:34.347 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:01.811 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:11:34.367 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:01.822 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:34.395 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:10.169 INFO:tasks.ceph.osd.6.smithi083.stderr:*** Caught signal (Aborted) ** 2024-03-28T16:26:10.169 INFO:tasks.ceph.osd.6.smithi083.stderr: in thread 29298640 thread_name:tp_osd_tp 2024-03-28T16:26:10.190 INFO:tasks.workunit.client.0.smithi007.stdout:eWrite 2024-03-28T16:26:10.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (52776 ms) 2024-03-28T16:26:10.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-03-28T16:26:10.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (63241 ms) 2024-03-28T16:26:10.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-03-28T16:26:10.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7278 ms) 2024-03-28T16:26:10.191 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-03-28T16:26:10.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6996 ms) 2024-03-28T16:26:10.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-03-28T16:26:10.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (6905 ms) 2024-03-28T16:26:10.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-03-28T16:26:10.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (8125 ms) 2024-03-28T16:26:10.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-03-28T16:26:10.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (8007 ms) 2024-03-28T16:26:10.192 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-03-28T16:26:10.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7033 ms) 2024-03-28T16:26:10.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-03-28T16:26:10.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7439 ms) 2024-03-28T16:26:10.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-03-28T16:26:10.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (85085 ms) 2024-03-28T16:26:10.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-03-28T16:26:10.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (12029 ms) 2024-03-28T16:26:10.193 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-03-28T16:26:10.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7540 ms) 2024-03-28T16:26:10.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-03-28T16:26:10.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (21187 ms) 2024-03-28T16:26:10.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-03-28T16:26:10.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6392 ms) 2024-03-28T16:26:10.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-03-28T16:26:10.194 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7052 ms) 2024-03-28T16:26:10.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-03-28T16:26:10.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7399 ms) 2024-03-28T16:26:10.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (314484 ms total) 2024-03-28T16:26:10.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: 2024-03-28T16:26:10.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [----------] Global test environment tear-down 2024-03-28T16:26:10.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (563240 ms total) 2024-03-28T16:26:10.195 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio: [ PASSED ] 42 tests. 2024-03-28T16:26:10.246 INFO:tasks.ceph.osd.6.smithi083.stderr: ceph version 18.2.2-1053-ge58b9979 (e58b9979847f029f7ab1514b5f5f99b9f03d97f6) reef (stable) 2024-03-28T16:26:10.247 INFO:tasks.ceph.osd.6.smithi083.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x549fdb0] 2024-03-28T16:26:10.247 INFO:tasks.ceph.osd.6.smithi083.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x54ec54c] 2024-03-28T16:26:10.247 INFO:tasks.ceph.osd.6.smithi083.stderr: 3: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point > >)+0x26f) [0xb4f56f] 2024-03-28T16:26:10.247 INFO:tasks.ceph.osd.6.smithi083.stderr: 4: (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, std::chrono::duration >, std::chrono::duration >)+0x74) [0xb4f664] 2024-03-28T16:26:10.247 INFO:tasks.ceph.osd.6.smithi083.stderr: 5: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x290) [0xb53130] 2024-03-28T16:26:10.247 INFO:tasks.ceph.osd.6.smithi083.stderr: 6: ceph-osd(+0xa4b6e4) [0xb536e4] 2024-03-28T16:26:10.247 INFO:tasks.ceph.osd.6.smithi083.stderr: 7: /lib64/libc.so.6(+0x9f802) [0x54ea802] 2024-03-28T16:26:10.247 INFO:tasks.ceph.osd.6.smithi083.stderr: 8: clone() 2024-03-28T16:26:10.335 INFO:tasks.ceph.osd.6.smithi083.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2-1053-ge58b9979/rpm/el9/BUILD/ceph-18.2.2-1053-ge58b9979/src/common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(const ceph::heartbeat_handle_d*, const char*, ceph::coarse_mono_time)' thread 2daa1640 time 2024-03-28T16:26:10.318659+0000 2024-03-28T16:26:10.335 INFO:tasks.ceph.osd.6.smithi083.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2-1053-ge58b9979/rpm/el9/BUILD/ceph-18.2.2-1053-ge58b9979/src/common/HeartbeatMap.cc: 85: ceph_abort_msg("hit suicide timeout") 2024-03-28T16:26:10.339 INFO:tasks.ceph.osd.6.smithi083.stderr: ceph version 18.2.2-1053-ge58b9979 (e58b9979847f029f7ab1514b5f5f99b9f03d97f6) reef (stable) 2024-03-28T16:26:10.339 INFO:tasks.ceph.osd.6.smithi083.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x4d2984] 2024-03-28T16:26:10.339 INFO:tasks.ceph.osd.6.smithi083.stderr: 2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point > >)+0x2a3) [0xb4f5a3] 2024-03-28T16:26:10.339 INFO:tasks.ceph.osd.6.smithi083.stderr: 3: (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, std::chrono::duration >, std::chrono::duration >)+0x74) [0xb4f664] 2024-03-28T16:26:10.339 INFO:tasks.ceph.osd.6.smithi083.stderr: 4: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x290) [0xb53130] 2024-03-28T16:26:10.339 INFO:tasks.ceph.osd.6.smithi083.stderr: 5: ceph-osd(+0xa4b6e4) [0xb536e4] 2024-03-28T16:26:10.340 INFO:tasks.ceph.osd.6.smithi083.stderr: 6: /lib64/libc.so.6(+0x9f802) [0x54ea802] 2024-03-28T16:26:10.340 INFO:tasks.ceph.osd.6.smithi083.stderr: 7: clone() 2024-03-28T16:26:11.799 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:44.371 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:11.854 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:44.427 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:11.856 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:44.429 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:11.857 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:11:44.414 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:11.859 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:11:44.416 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:11.874 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:44.447 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:11.893 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:44.466 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:11.896 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:11:44.452 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:11.922 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:44.495 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:13.194 INFO:tasks.ceph.osd.6.smithi083.stderr:==00:00:11:23.279 35297== 2024-03-28T16:26:13.194 INFO:tasks.ceph.osd.6.smithi083.stderr:==00:00:11:23.279 35297== Exit program on first error (--exit-on-first-error=yes) 2024-03-28T16:26:13.335 INFO:tasks.ceph.osd.6.smithi083.stderr:daemon-helper: command failed with exit status 42 2024-03-28T16:26:14.210 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-03-28T16:26:20.420 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~6s 2024-03-28T16:26:21.936 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:54.508 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:21.938 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:54.511 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:21.976 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:54.549 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:21.984 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:11:54.540 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:21.988 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:11:54.545 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:21.993 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:54.566 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:21.997 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:54.570 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:22.016 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:11:54.573 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:22.026 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:54.599 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:26.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~12s 2024-03-28T16:26:26.801 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:59.374 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:26.803 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:59.376 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:26.819 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:59.391 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:26.851 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:11:59.407 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:26.852 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:59.426 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:26.854 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:59.427 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:26.869 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:11:59.426 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:26.870 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:59.442 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:26.883 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:11:59.440 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:32.839 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~19s 2024-03-28T16:26:39.050 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~25s 2024-03-28T16:26:41.960 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:14.532 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:41.963 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:14.536 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:42.003 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:14.575 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:42.010 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:12:14.566 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:42.011 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:12:14.567 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:42.021 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:14.591 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:42.022 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:14.593 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:42.041 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:12:14.597 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:42.052 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:14.625 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:45.260 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~31s 2024-03-28T16:26:51.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~37s 2024-03-28T16:26:56.931 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:29.504 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:56.933 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:29.507 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:56.961 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:29.534 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:56.980 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:12:29.536 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:56.983 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:12:29.539 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:56.985 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:29.558 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:56.987 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:29.560 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:57.007 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:12:29.564 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:57.013 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:29.586 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:26:57.681 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~43s 2024-03-28T16:27:00.142 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:32.715 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:00.144 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:32.717 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:00.153 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:12:32.710 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:00.154 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:32.728 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:00.196 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:12:32.752 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:00.198 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:32.771 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:00.216 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:12:32.772 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:00.217 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:32.791 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:00.221 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:32.794 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:01.951 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:34.524 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:01.953 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:34.526 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:01.966 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:34.539 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:01.999 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:12:34.555 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:02.002 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:12:34.558 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:02.005 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:34.578 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:02.008 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:34.581 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:02.010 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:12:34.567 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:02.017 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:34.590 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:03.890 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~50s 2024-03-28T16:27:10.100 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~56s 2024-03-28T16:27:11.991 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:44.563 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:11.995 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:44.568 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:12.022 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:44.595 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:12.041 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:44.614 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:12.041 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:12:44.598 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:12.043 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:12:44.600 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:12.046 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:44.619 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:12.061 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:12:44.617 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:12.072 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:44.645 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:16.311 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~62s 2024-03-28T16:27:21.972 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:54.545 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:22.025 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:54.598 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:22.045 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:54.618 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:22.072 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:12:54.628 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:22.072 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:54.645 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:22.074 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:12:54.630 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:22.076 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:54.649 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:22.090 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:12:54.646 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:22.104 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:54.677 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:22.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~68s 2024-03-28T16:27:22.665 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:12:55.221 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:22.666 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:12:55.223 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:22.675 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:12:55.231 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:22.677 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:55.250 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:22.678 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:55.251 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:22.686 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:55.259 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:22.722 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:55.295 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:22.723 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:55.296 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:22.735 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:55.308 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:28.730 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~75s 2024-03-28T16:27:34.940 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~81s 2024-03-28T16:27:37.067 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:09.639 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:37.116 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:09.689 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:37.136 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:09.708 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:37.164 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:13:09.721 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:37.165 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:13:09.722 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:37.166 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:09.740 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:37.168 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:09.741 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:37.183 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:13:09.739 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:37.196 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:09.769 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:41.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~87s 2024-03-28T16:27:42.076 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:14.649 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:42.124 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:14.697 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:42.137 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:14.710 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:42.169 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:13:14.725 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:42.170 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:13:14.727 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:42.180 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:14.753 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:42.181 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:14.754 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:42.181 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:13:14.738 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:42.193 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:14.765 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:27:47.360 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~93s 2024-03-28T16:27:53.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~99s 2024-03-28T16:27:59.779 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~106s 2024-03-28T16:28:00.009 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:28:00.003+0000 ea59640 -1 osd.4 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:34.468141+0000 front 2024-03-28T16:26:34.470288+0000 (oldest deadline 2024-03-28T16:27:59.146247+0000) 2024-03-28T16:28:00.520 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:28:00.519+0000 ea59640 -1 osd.7 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:37.718028+0000 front 2024-03-28T16:26:37.716799+0000 (oldest deadline 2024-03-28T16:28:00.010783+0000) 2024-03-28T16:28:01.009 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:28:01.006+0000 ea59640 -1 osd.4 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:34.468141+0000 front 2024-03-28T16:26:34.470288+0000 (oldest deadline 2024-03-28T16:27:59.146247+0000) 2024-03-28T16:28:01.172 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-03-28T16:28:01.171+0000 ea59640 -1 osd.0 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:37.934218+0000 front 2024-03-28T16:26:37.929708+0000 (oldest deadline 2024-03-28T16:28:00.224386+0000) 2024-03-28T16:28:01.530 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:28:01.529+0000 ea59640 -1 osd.7 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:37.718028+0000 front 2024-03-28T16:26:37.716799+0000 (oldest deadline 2024-03-28T16:28:00.010783+0000) 2024-03-28T16:28:01.603 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:28:01.600+0000 ea59640 -1 osd.3 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:36.051541+0000 front 2024-03-28T16:26:36.053552+0000 (oldest deadline 2024-03-28T16:28:00.727039+0000) 2024-03-28T16:28:02.043 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:28:02.043+0000 ea59640 -1 osd.4 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:34.468141+0000 front 2024-03-28T16:26:34.470288+0000 (oldest deadline 2024-03-28T16:27:59.146247+0000) 2024-03-28T16:28:02.136 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:34.709 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:02.161 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-03-28T16:28:02.159+0000 ea59640 -1 osd.0 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:37.934218+0000 front 2024-03-28T16:26:37.929708+0000 (oldest deadline 2024-03-28T16:28:00.224386+0000) 2024-03-28T16:28:02.185 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:13:34.741 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:02.186 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:13:34.743 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:02.191 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:34.764 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:02.209 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:13:34.765 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:02.218 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:34.791 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:02.243 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:34.815 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:02.244 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:34.817 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:02.267 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:34.840 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:02.563 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:28:02.562+0000 ea59640 -1 osd.3 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:36.051541+0000 front 2024-03-28T16:26:36.053552+0000 (oldest deadline 2024-03-28T16:28:00.727039+0000) 2024-03-28T16:28:02.576 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:28:02.574+0000 ea59640 -1 osd.7 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:37.718028+0000 front 2024-03-28T16:26:37.716799+0000 (oldest deadline 2024-03-28T16:28:00.010783+0000) 2024-03-28T16:28:03.077 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:28:03.077+0000 ea59640 -1 osd.4 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:34.468141+0000 front 2024-03-28T16:26:34.470288+0000 (oldest deadline 2024-03-28T16:27:59.146247+0000) 2024-03-28T16:28:03.169 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-03-28T16:28:03.168+0000 ea59640 -1 osd.0 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:37.934218+0000 front 2024-03-28T16:26:37.929708+0000 (oldest deadline 2024-03-28T16:28:00.224386+0000) 2024-03-28T16:28:03.296 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:28:03.295+0000 12a61640 -1 osd.5 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:37.196564+0000 front 2024-03-28T16:26:37.195731+0000 (oldest deadline 2024-03-28T16:28:03.089661+0000) 2024-03-28T16:28:03.447 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:28:03.445+0000 ea59640 -1 osd.2 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:37.368233+0000 front 2024-03-28T16:26:37.338830+0000 (oldest deadline 2024-03-28T16:28:03.177047+0000) 2024-03-28T16:28:03.524 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:28:03.523+0000 ea59640 -1 osd.3 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:36.051541+0000 front 2024-03-28T16:26:36.053552+0000 (oldest deadline 2024-03-28T16:28:00.727039+0000) 2024-03-28T16:28:03.606 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:28:03.605+0000 ea59640 -1 osd.7 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:37.718028+0000 front 2024-03-28T16:26:37.716799+0000 (oldest deadline 2024-03-28T16:28:00.010783+0000) 2024-03-28T16:28:04.088 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:28:04.082+0000 ea59640 -1 osd.4 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:34.468141+0000 front 2024-03-28T16:26:34.470288+0000 (oldest deadline 2024-03-28T16:27:59.146247+0000) 2024-03-28T16:28:04.169 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-03-28T16:28:04.164+0000 ea59640 -1 osd.0 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:37.934218+0000 front 2024-03-28T16:26:37.929708+0000 (oldest deadline 2024-03-28T16:28:00.224386+0000) 2024-03-28T16:28:04.301 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:28:04.300+0000 12a61640 -1 osd.5 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:37.196564+0000 front 2024-03-28T16:26:37.195731+0000 (oldest deadline 2024-03-28T16:28:03.089661+0000) 2024-03-28T16:28:04.456 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:28:04.455+0000 ea59640 -1 osd.2 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:37.368233+0000 front 2024-03-28T16:26:37.338830+0000 (oldest deadline 2024-03-28T16:28:03.177047+0000) 2024-03-28T16:28:04.505 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:28:04.503+0000 ea59640 -1 osd.3 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:36.051541+0000 front 2024-03-28T16:26:36.053552+0000 (oldest deadline 2024-03-28T16:28:00.727039+0000) 2024-03-28T16:28:04.649 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:28:04.647+0000 ea59640 -1 osd.7 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:37.718028+0000 front 2024-03-28T16:26:37.716799+0000 (oldest deadline 2024-03-28T16:28:00.010783+0000) 2024-03-28T16:28:05.121 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-03-28T16:28:05.120+0000 ea59640 -1 osd.0 390 heartbeat_check: no reply from 172.21.15.7:6804 osd.1 since back 2024-03-28T16:26:37.934218+0000 front 2024-03-28T16:26:37.929708+0000 (oldest deadline 2024-03-28T16:28:00.224386+0000) 2024-03-28T16:28:05.989 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~112s 2024-03-28T16:28:12.202 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~118s 2024-03-28T16:28:12.291 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:44.863 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:12.293 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:44.866 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:12.330 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:44.903 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:12.341 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:13:44.897 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:12.342 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:13:44.898 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:12.363 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:13:44.919 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:12.370 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:44.943 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:12.372 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:44.945 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:12.407 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:44.980 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:18.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~124s 2024-03-28T16:28:22.327 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:54.899 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:22.390 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:54.963 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:22.417 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:54.988 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:22.437 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:13:54.993 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:22.437 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:13:54.994 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:22.438 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:55.011 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:22.441 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:55.014 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:22.464 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:13:55.021 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:22.470 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:13:55.043 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:24.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~130s 2024-03-28T16:28:27.370 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:59.942 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:27.421 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:13:59.994 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:27.433 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:00.006 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:27.473 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:00.029 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:27.474 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:00.030 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:27.477 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:00.050 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:27.481 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:00.054 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:27.489 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:00.045 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:27.494 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:00.067 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:30.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~137s 2024-03-28T16:28:37.043 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~143s 2024-03-28T16:28:42.450 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:15.022 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:42.451 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:15.024 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:42.472 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:15.045 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:42.501 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:15.074 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:42.502 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:15.058 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:42.503 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:15.060 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:42.507 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:15.080 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:42.523 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:15.079 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:42.538 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:15.111 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:43.187 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:15.743 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:43.187 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:15.744 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:43.195 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:15.752 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:43.209 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:15.782 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:43.231 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:15.804 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:43.243 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:15.816 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:43.253 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~149s 2024-03-28T16:28:43.295 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:15.868 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:43.296 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:15.869 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:43.304 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:15.877 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:49.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~155s 2024-03-28T16:28:52.578 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:25.151 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:52.581 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:25.154 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:52.597 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:25.170 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:52.624 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:25.180 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:52.627 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:25.183 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:52.631 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:25.204 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:52.633 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:25.206 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:52.642 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:25.198 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:52.650 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:25.223 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:28:55.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~161s 2024-03-28T16:29:02.244 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~168s 2024-03-28T16:29:02.510 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:35.082 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:02.564 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:35.137 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:02.585 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:35.158 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:02.616 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:35.172 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:02.619 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:35.175 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:02.629 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:35.202 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:02.631 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:35.204 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:02.635 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:35.191 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:02.647 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:35.220 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:07.529 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:40.102 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:07.578 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:40.134 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:07.578 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:40.151 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:07.590 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:40.147 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:07.604 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:40.161 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:07.605 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:40.178 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:07.633 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:40.206 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:07.634 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:40.207 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:07.649 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:40.222 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:08.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~174s 2024-03-28T16:29:14.664 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~180s 2024-03-28T16:29:20.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~187s 2024-03-28T16:29:22.621 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:55.193 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:22.623 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:55.196 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:22.658 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:55.231 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:22.674 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:55.230 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:22.678 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:55.234 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:22.698 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:55.271 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:22.701 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:55.257 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:22.702 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:55.275 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:22.734 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:55.307 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:23.383 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:55.939 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:23.384 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:55.940 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:23.392 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:14:55.948 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:23.487 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:56.060 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:23.489 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:56.063 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:23.501 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:14:56.074 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:23.529 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:56.101 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:23.529 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:56.103 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:23.538 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:14:56.111 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:27.085 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~193s 2024-03-28T16:29:33.294 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~199s 2024-03-28T16:29:36.632 INFO:tasks.ceph.osd.1.smithi007.stderr:*** Caught signal (Aborted) ** 2024-03-28T16:29:36.632 INFO:tasks.ceph.osd.1.smithi007.stderr: in thread 28296640 thread_name:tp_osd_tp 2024-03-28T16:29:36.685 INFO:tasks.ceph.osd.1.smithi007.stderr: ceph version 18.2.2-1053-ge58b9979 (e58b9979847f029f7ab1514b5f5f99b9f03d97f6) reef (stable) 2024-03-28T16:29:36.685 INFO:tasks.ceph.osd.1.smithi007.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x549fdb0] 2024-03-28T16:29:36.685 INFO:tasks.ceph.osd.1.smithi007.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x54ec54c] 2024-03-28T16:29:36.685 INFO:tasks.ceph.osd.1.smithi007.stderr: 3: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point > >)+0x26f) [0xb4f56f] 2024-03-28T16:29:36.685 INFO:tasks.ceph.osd.1.smithi007.stderr: 4: (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, std::chrono::duration >, std::chrono::duration >)+0x74) [0xb4f664] 2024-03-28T16:29:36.685 INFO:tasks.ceph.osd.1.smithi007.stderr: 5: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x290) [0xb53130] 2024-03-28T16:29:36.685 INFO:tasks.ceph.osd.1.smithi007.stderr: 6: ceph-osd(+0xa4b6e4) [0xb536e4] 2024-03-28T16:29:36.685 INFO:tasks.ceph.osd.1.smithi007.stderr: 7: /lib64/libc.so.6(+0x9f802) [0x54ea802] 2024-03-28T16:29:36.686 INFO:tasks.ceph.osd.1.smithi007.stderr: 8: clone() 2024-03-28T16:29:36.756 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-03-28T16:29:36.740+0000 28296640 -1 *** Caught signal (Aborted) ** 2024-03-28T16:29:36.756 INFO:tasks.ceph.osd.1.smithi007.stderr: in thread 28296640 thread_name:tp_osd_tp 2024-03-28T16:29:36.756 INFO:tasks.ceph.osd.1.smithi007.stderr: 2024-03-28T16:29:36.756 INFO:tasks.ceph.osd.1.smithi007.stderr: ceph version 18.2.2-1053-ge58b9979 (e58b9979847f029f7ab1514b5f5f99b9f03d97f6) reef (stable) 2024-03-28T16:29:36.757 INFO:tasks.ceph.osd.1.smithi007.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x549fdb0] 2024-03-28T16:29:36.757 INFO:tasks.ceph.osd.1.smithi007.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x54ec54c] 2024-03-28T16:29:36.757 INFO:tasks.ceph.osd.1.smithi007.stderr: 3: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point > >)+0x26f) [0xb4f56f] 2024-03-28T16:29:36.757 INFO:tasks.ceph.osd.1.smithi007.stderr: 4: (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, std::chrono::duration >, std::chrono::duration >)+0x74) [0xb4f664] 2024-03-28T16:29:36.757 INFO:tasks.ceph.osd.1.smithi007.stderr: 5: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x290) [0xb53130] 2024-03-28T16:29:36.757 INFO:tasks.ceph.osd.1.smithi007.stderr: 6: ceph-osd(+0xa4b6e4) [0xb536e4] 2024-03-28T16:29:36.757 INFO:tasks.ceph.osd.1.smithi007.stderr: 7: /lib64/libc.so.6(+0x9f802) [0x54ea802] 2024-03-28T16:29:36.757 INFO:tasks.ceph.osd.1.smithi007.stderr: 8: clone() 2024-03-28T16:29:36.757 INFO:tasks.ceph.osd.1.smithi007.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T16:29:36.757 INFO:tasks.ceph.osd.1.smithi007.stderr: 2024-03-28T16:29:37.190 INFO:tasks.ceph.osd.1.smithi007.stderr: -51> 2024-03-28T16:29:36.740+0000 28296640 -1 *** Caught signal (Aborted) ** 2024-03-28T16:29:37.191 INFO:tasks.ceph.osd.1.smithi007.stderr: in thread 28296640 thread_name:tp_osd_tp 2024-03-28T16:29:37.191 INFO:tasks.ceph.osd.1.smithi007.stderr: 2024-03-28T16:29:37.191 INFO:tasks.ceph.osd.1.smithi007.stderr: ceph version 18.2.2-1053-ge58b9979 (e58b9979847f029f7ab1514b5f5f99b9f03d97f6) reef (stable) 2024-03-28T16:29:37.191 INFO:tasks.ceph.osd.1.smithi007.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x549fdb0] 2024-03-28T16:29:37.191 INFO:tasks.ceph.osd.1.smithi007.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x54ec54c] 2024-03-28T16:29:37.191 INFO:tasks.ceph.osd.1.smithi007.stderr: 3: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point > >)+0x26f) [0xb4f56f] 2024-03-28T16:29:37.191 INFO:tasks.ceph.osd.1.smithi007.stderr: 4: (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, std::chrono::duration >, std::chrono::duration >)+0x74) [0xb4f664] 2024-03-28T16:29:37.191 INFO:tasks.ceph.osd.1.smithi007.stderr: 5: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x290) [0xb53130] 2024-03-28T16:29:37.191 INFO:tasks.ceph.osd.1.smithi007.stderr: 6: ceph-osd(+0xa4b6e4) [0xb536e4] 2024-03-28T16:29:37.191 INFO:tasks.ceph.osd.1.smithi007.stderr: 7: /lib64/libc.so.6(+0x9f802) [0x54ea802] 2024-03-28T16:29:37.191 INFO:tasks.ceph.osd.1.smithi007.stderr: 8: clone() 2024-03-28T16:29:37.191 INFO:tasks.ceph.osd.1.smithi007.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T16:29:37.191 INFO:tasks.ceph.osd.1.smithi007.stderr: 2024-03-28T16:29:37.610 INFO:tasks.ceph.osd.1.smithi007.stderr: -260> 2024-03-28T16:29:36.740+0000 28296640 -1 *** Caught signal (Aborted) ** 2024-03-28T16:29:37.610 INFO:tasks.ceph.osd.1.smithi007.stderr: in thread 28296640 thread_name:tp_osd_tp 2024-03-28T16:29:37.611 INFO:tasks.ceph.osd.1.smithi007.stderr: 2024-03-28T16:29:37.611 INFO:tasks.ceph.osd.1.smithi007.stderr: ceph version 18.2.2-1053-ge58b9979 (e58b9979847f029f7ab1514b5f5f99b9f03d97f6) reef (stable) 2024-03-28T16:29:37.611 INFO:tasks.ceph.osd.1.smithi007.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x549fdb0] 2024-03-28T16:29:37.611 INFO:tasks.ceph.osd.1.smithi007.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x54ec54c] 2024-03-28T16:29:37.611 INFO:tasks.ceph.osd.1.smithi007.stderr: 3: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point > >)+0x26f) [0xb4f56f] 2024-03-28T16:29:37.611 INFO:tasks.ceph.osd.1.smithi007.stderr: 4: (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, std::chrono::duration >, std::chrono::duration >)+0x74) [0xb4f664] 2024-03-28T16:29:37.611 INFO:tasks.ceph.osd.1.smithi007.stderr: 5: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x290) [0xb53130] 2024-03-28T16:29:37.611 INFO:tasks.ceph.osd.1.smithi007.stderr: 6: ceph-osd(+0xa4b6e4) [0xb536e4] 2024-03-28T16:29:37.611 INFO:tasks.ceph.osd.1.smithi007.stderr: 7: /lib64/libc.so.6(+0x9f802) [0x54ea802] 2024-03-28T16:29:37.611 INFO:tasks.ceph.osd.1.smithi007.stderr: 8: clone() 2024-03-28T16:29:37.611 INFO:tasks.ceph.osd.1.smithi007.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-28T16:29:37.611 INFO:tasks.ceph.osd.1.smithi007.stderr: 2024-03-28T16:29:37.666 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:10.239 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:37.670 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:10.243 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:37.697 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:10.270 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:37.718 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:15:10.274 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:37.720 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:15:10.276 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:37.721 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:10.294 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:37.723 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:10.296 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:37.745 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:15:10.301 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:37.763 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:10.336 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:39.505 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~205s 2024-03-28T16:29:40.047 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:14:50.156 34919== 2024-03-28T16:29:40.047 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:14:50.156 34919== Exit program on first error (--exit-on-first-error=yes) 2024-03-28T16:29:40.214 INFO:tasks.ceph.osd.1.smithi007.stderr:daemon-helper: command failed with exit status 42 2024-03-28T16:29:42.634 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:15.207 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:42.683 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:15.256 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:42.695 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:15.268 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:42.727 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:15:15.284 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:42.730 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:15:15.287 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:42.738 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:15.311 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:42.739 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:15.312 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:42.741 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:15:15.298 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:42.750 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:15.323 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:45.615 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-03-28T16:29:45.615 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~212s 2024-03-28T16:29:51.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~6s 2024-03-28T16:29:51.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~218s 2024-03-28T16:29:52.711 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:25.284 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:52.713 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:25.286 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:52.730 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:25.303 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:52.758 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:15:25.314 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:52.761 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:15:25.317 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:52.763 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:25.336 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:52.765 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:25.338 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:52.777 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:15:25.334 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:52.782 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:25.355 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:29:57.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~12s 2024-03-28T16:29:57.834 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~224s 2024-03-28T16:30:02.740 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:35.313 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:02.742 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:35.315 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:02.758 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:35.331 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:02.789 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:15:35.345 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:02.790 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:15:35.347 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:02.794 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:35.367 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:02.796 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:35.369 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:02.805 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:15:35.362 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:02.812 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:35.385 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:03.942 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~18s 2024-03-28T16:30:03.942 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~230s 2024-03-28T16:30:07.774 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:40.347 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:07.776 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:40.349 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:07.788 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:40.361 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:07.819 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:15:40.375 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:07.822 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:15:40.378 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:07.827 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:40.400 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:07.828 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:40.401 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:07.832 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:15:40.389 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:07.839 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:40.413 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:10.052 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~24s 2024-03-28T16:30:10.052 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~236s 2024-03-28T16:30:16.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~31s 2024-03-28T16:30:16.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~242s 2024-03-28T16:30:22.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~37s 2024-03-28T16:30:22.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~248s 2024-03-28T16:30:22.801 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:55.373 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:22.806 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:55.378 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:22.826 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:55.399 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:22.853 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:15:55.409 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:22.854 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:55.427 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:22.854 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:15:55.411 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:22.858 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:55.431 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:22.874 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:15:55.430 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:22.879 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:55.452 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:23.525 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:15:56.081 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:23.526 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:15:56.082 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:23.533 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:15:56.090 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:23.572 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:56.145 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:23.573 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:56.146 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:23.581 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:15:56.154 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:23.600 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:56.173 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:23.601 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:56.174 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:23.609 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:15:56.182 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:28.383 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~43s 2024-03-28T16:30:28.383 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~254s 2024-03-28T16:30:34.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~49s 2024-03-28T16:30:34.493 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~260s 2024-03-28T16:30:37.843 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:10.415 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:37.844 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:10.417 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:37.862 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:10.435 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:37.890 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:16:10.447 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:37.893 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:16:10.449 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:37.897 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:10.470 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:37.913 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:16:10.469 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:37.933 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:10.506 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:37.951 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:10.524 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:40.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~55s 2024-03-28T16:30:40.603 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~266s 2024-03-28T16:30:42.849 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:15.422 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:42.853 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:15.426 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:42.865 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:15.438 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:42.899 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:16:15.455 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:42.900 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:16:15.456 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:42.902 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:15.475 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:42.904 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:15.477 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:42.911 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:16:15.468 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:42.918 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:15.491 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:46.712 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~61s 2024-03-28T16:30:46.712 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~273s 2024-03-28T16:30:47.889 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:20.462 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:47.891 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:20.464 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:47.911 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:20.484 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:47.936 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:16:20.492 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:47.939 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:16:20.495 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:47.946 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:20.519 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:47.953 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:16:20.509 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:47.964 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:20.537 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:47.980 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:20.553 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:30:52.822 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~67s 2024-03-28T16:30:52.822 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~279s 2024-03-28T16:30:58.932 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~73s 2024-03-28T16:30:58.932 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~285s 2024-03-28T16:31:02.929 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:35.502 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:02.930 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:35.504 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:02.949 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:35.522 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:02.974 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:16:35.531 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:02.976 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:16:35.532 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:02.983 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:35.555 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:02.984 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:35.557 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:02.994 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:16:35.550 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:03.003 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:35.576 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:03.582 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:16:36.139 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:03.583 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:16:36.140 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:03.591 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:16:36.147 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:03.617 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:36.190 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:03.618 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:36.191 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:03.626 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:36.199 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:03.635 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:36.208 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:03.636 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:36.210 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:03.645 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:36.218 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:05.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~79s 2024-03-28T16:31:05.043 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~291s 2024-03-28T16:31:11.152 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~86s 2024-03-28T16:31:11.152 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~297s 2024-03-28T16:31:17.262 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~92s 2024-03-28T16:31:17.262 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~303s 2024-03-28T16:31:17.262 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-03-28T16:31:17.958 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:50.531 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:17.962 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:50.536 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:17.982 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:50.554 34605== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:18.012 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:16:50.568 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:18.013 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:16:50.569 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:18.013 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:50.586 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:18.015 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:50.588 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:18.031 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:16:50.588 34751== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:18.034 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:16:50.607 34606== Warning: unimplemented fcntl command: 1036 2024-03-28T16:31:18.370 INFO:tasks.ceph.osd.0:Sent signal 15 2024-03-28T16:31:18.371 INFO:tasks.ceph.osd.2:Sent signal 15 2024-03-28T16:31:18.371 INFO:tasks.ceph.osd.3:Sent signal 15 2024-03-28T16:31:18.371 INFO:tasks.ceph.osd.4:Sent signal 15 2024-03-28T16:31:18.372 INFO:tasks.ceph.osd.5:Sent signal 15 2024-03-28T16:31:18.372 INFO:tasks.ceph.osd.7:Sent signal 15 2024-03-28T16:31:18.372 INFO:tasks.ceph.mon.a:Sent signal 15 2024-03-28T16:31:18.372 INFO:tasks.ceph.mon.c:Sent signal 15 2024-03-28T16:31:18.373 INFO:tasks.ceph.mon.b:Sent signal 15 2024-03-28T16:31:18.373 INFO:tasks.ceph.mgr.y:Sent signal 15 2024-03-28T16:31:18.373 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-03-28T16:31:18.374 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-03-28T16:31:18.373+0000 7efddcaa9640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34749) UID: 0 2024-03-28T16:31:18.384 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:31:18.375+0000 b47f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35292) UID: 0 2024-03-28T16:31:18.385 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:31:18.375+0000 b47f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35288) UID: 0 2024-03-28T16:31:18.385 INFO:tasks.ceph.osd.5.smithi083.stderr:2024-03-28T16:31:18.376+0000 b47f640 -1 osd.5 406 *** Got signal Terminated *** 2024-03-28T16:31:18.386 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-03-28T16:31:18.375+0000 b47f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 34892) UID: 0 2024-03-28T16:31:18.386 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-03-28T16:31:18.376+0000 b47f640 -1 osd.0 406 *** Got signal Terminated *** 2024-03-28T16:31:18.388 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:31:18.376+0000 b47f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 34902) UID: 0 2024-03-28T16:31:18.388 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-28T16:31:18.378+0000 b47f640 -1 osd.2 406 *** Got signal Terminated *** 2024-03-28T16:31:18.389 INFO:tasks.ceph.osd.7.smithi083.stderr:2024-03-28T16:31:18.376+0000 b47f640 -1 osd.7 406 *** Got signal Terminated *** 2024-03-28T16:31:18.389 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:31:18.376+0000 b47f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35284) UID: 0 2024-03-28T16:31:18.390 INFO:tasks.ceph.osd.4.smithi083.stderr:2024-03-28T16:31:18.378+0000 b47f640 -1 osd.4 406 *** Got signal Terminated *** 2024-03-28T16:31:18.394 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:31:18.376+0000 b47f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 34912) UID: 0 2024-03-28T16:31:18.394 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-28T16:31:18.378+0000 b47f640 -1 osd.3 406 *** Got signal Terminated *** 2024-03-28T16:31:18.395 INFO:tasks.ceph.mon.c.smithi007.stderr:2024-03-28T16:31:18.381+0000 9f50640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 34574) UID: 0 2024-03-28T16:31:18.396 INFO:tasks.ceph.mon.a.smithi007.stderr:2024-03-28T16:31:18.381+0000 9f50640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 34572) UID: 0 2024-03-28T16:31:18.398 INFO:tasks.ceph.mon.c.smithi007.stderr:2024-03-28T16:31:18.383+0000 9f50640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-28T16:31:18.399 INFO:tasks.ceph.mon.a.smithi007.stderr:2024-03-28T16:31:18.383+0000 9f50640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-28T16:31:18.400 INFO:tasks.ceph.mon.b.smithi083.stderr:2024-03-28T16:31:18.382+0000 9f50640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34747) UID: 0 2024-03-28T16:31:18.401 INFO:tasks.ceph.mon.b.smithi083.stderr:2024-03-28T16:31:18.385+0000 9f50640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-03-28T16:31:18.574 INFO:tasks.ceph.mgr.y.smithi007.stderr:daemon-helper: command crashed with signal 15 2024-03-28T16:31:19.197 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:16:51.754 34751== 2024-03-28T16:31:19.198 INFO:tasks.ceph.mon.b.smithi083.stderr:==00:00:16:51.754 34751== Exit program on first error (--exit-on-first-error=yes) 2024-03-28T16:31:19.375 INFO:tasks.ceph.mon.b.smithi083.stderr:daemon-helper: command failed with exit status 42 2024-03-28T16:31:19.445 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:52.017 34605== 2024-03-28T16:31:19.445 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:16:52.017 34605== Exit program on first error (--exit-on-first-error=yes) 2024-03-28T16:31:19.574 INFO:tasks.ceph.mon.a.smithi007.stderr:daemon-helper: command failed with exit status 42 2024-03-28T16:31:50.291 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:22.864 34606== 2024-03-28T16:31:50.292 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:17:22.864 34606== Exit program on first error (--exit-on-first-error=yes) 2024-03-28T16:31:50.405 INFO:tasks.ceph.mon.c.smithi007.stderr:daemon-helper: command failed with exit status 42 2024-03-28T17:06:38.017 INFO:tasks.workunit.client.0.smithi007.stdout:dosAio.RoundTripAppendPP (56603 ms) 2024-03-28T17:06:38.017 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-03-28T17:06:38.017 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3328 ms) 2024-03-28T17:06:38.017 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-03-28T17:06:38.017 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (60207 ms) 2024-03-28T17:06:38.017 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-03-28T17:06:38.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (13000 ms) 2024-03-28T17:06:38.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-03-28T17:06:38.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: setting pool EIO 2024-03-28T17:06:38.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: max_success 37, min_failed 38 2024-03-28T17:06:38.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (6025 ms) 2024-03-28T17:06:38.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-03-28T17:06:38.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3942 ms) 2024-03-28T17:06:38.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (385951 ms total) 2024-03-28T17:06:38.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: 2024-03-28T17:06:38.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-03-28T17:06:38.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-03-28T17:06:38.018 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3395 ms) 2024-03-28T17:06:38.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-03-28T17:06:38.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9648 ms) 2024-03-28T17:06:38.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-03-28T17:06:38.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (15652 ms) 2024-03-28T17:06:38.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-03-28T17:06:38.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (2884 ms) 2024-03-28T17:06:38.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (31579 ms total) 2024-03-28T17:06:38.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: 2024-03-28T17:06:38.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-03-28T17:06:38.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-28T17:06:38.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (2805 ms) 2024-03-28T17:06:38.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (2805 ms total) 2024-03-28T17:06:38.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: 2024-03-28T17:06:38.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-03-28T17:06:38.019 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-03-28T17:06:38.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (87892 ms) 2024-03-28T17:06:38.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-03-28T17:06:38.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (34213 ms) 2024-03-28T17:06:38.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-03-28T17:06:38.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6342 ms) 2024-03-28T17:06:38.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-03-28T17:06:38.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7035 ms) 2024-03-28T17:06:38.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-03-28T17:06:38.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3214 ms) 2024-03-28T17:06:38.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-03-28T17:06:38.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6918 ms) 2024-03-28T17:06:38.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-03-28T17:06:38.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7569 ms) 2024-03-28T17:06:38.020 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-03-28T17:06:38.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7146 ms) 2024-03-28T17:06:38.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-03-28T17:06:38.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7702 ms) 2024-03-28T17:06:38.021 INFO:tasks.workunit.client.0.smithi007.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-03-28T17:06:38.021 INFO:tasks.workunit.client.0.smithi007.stderr:bash: line 1: 40164 Alarm clock ceph_test_rados_api_aio_pp 2>&1 2024-03-28T17:06:38.021 INFO:tasks.workunit.client.0.smithi007.stderr: 40166 Done | tee ceph_test_rados_api_aio_pp.log 2024-03-28T17:06:38.021 INFO:tasks.workunit.client.0.smithi007.stderr: 40168 Done | sed "s/^/ api_aio_pp: /" 2024-03-28T22:16:46.912 INFO:tasks.workunit.client.0.smithi007.stderr:++ cleanup 2024-03-28T22:16:46.934 INFO:tasks.workunit.client.0.smithi007.stderr:++ pkill -P 40069 2024-03-28T22:16:46.936 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-28T22:16:46.936 INFO:tasks.workunit.client.0.smithi007.stderr:++ true 2024-03-28T22:16:46.936 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-03-28T22:16:46.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-28T22:16:47.223 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rados/test.sh) on smithi007 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e58b9979847f029f7ab1514b5f5f99b9f03d97f6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2024-03-28T22:16:47.702 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6eaa52a0fd0a46f8b253f613d5b82baf Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rados/test.sh) on smithi007 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e58b9979847f029f7ab1514b5f5f99b9f03d97f6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2024-03-28T22:16:47.705 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-28T22:16:47.714 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-28T22:16:47.714 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T22:16:47.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.817+0000 7fe57e2a4640 10 monclient: get_monmap_and_config 2024-03-28T22:16:47.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.817+0000 7fe57e2a4640 10 monclient: build_initial_monmap 2024-03-28T22:16:47.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.817+0000 7fe57e2a4640 10 monclient: monmap: 2024-03-28T22:16:47.820 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-28T22:16:47.821 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 808ff1a1-2b17-4c35-a89a-340dbbf674ee 2024-03-28T22:16:47.821 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-28T22:16:47.819191+0000 2024-03-28T22:16:47.821 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-28T22:16:47.819191+0000 2024-03-28T22:16:47.821 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-28T22:16:47.821 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-28T22:16:47.821 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2024-03-28T22:16:47.821 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.83:6789/0 mon.noname-c-legacy 2024-03-28T22:16:47.821 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2024-03-28T22:16:47.821 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2024-03-28T22:16:47.822 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.83:3300/0 mon.noname-c 2024-03-28T22:16:47.822 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-28T22:16:47.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.818+0000 7fe57e2a4640 1 Processor -- start 2024-03-28T22:16:47.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.818+0000 7fe57e2a4640 1 -- start start 2024-03-28T22:16:47.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.818+0000 7fe57e2a4640 10 monclient: init 2024-03-28T22:16:47.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.818+0000 7fe57e2a4640 10 monclient: _reopen_session rank -1 2024-03-28T22:16:47.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.818+0000 7fe57e2a4640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-03-28T22:16:47.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.818+0000 7fe57e2a4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe57806f600 addr v1:172.21.15.7:6789/0 2024-03-28T22:16:47.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.818+0000 7fe57e2a4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe57806f9e0 addr v1:172.21.15.83:6789/0 2024-03-28T22:16:47.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.818+0000 7fe57e2a4640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 0x7fe57814ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:47.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.818+0000 7fe57e2a4640 10 monclient(hunting): picked mon.noname-c con 0x7fe5780ad5d0 addr v2:172.21.15.83:3300/0 2024-03-28T22:16:47.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.818+0000 7fe57e2a4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780acf90 con 0x7fe57806f600 2024-03-28T22:16:47.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.818+0000 7fe57e2a4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780709c0 con 0x7fe57806f9e0 2024-03-28T22:16:47.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.818+0000 7fe57e2a4640 10 monclient(hunting): start opening mon connection 2024-03-28T22:16:47.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.818+0000 7fe57e2a4640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe5780ad5d0 2024-03-28T22:16:47.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.818+0000 7fe57e2a4640 10 monclient(hunting): _renew_subs 2024-03-28T22:16:47.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.818+0000 7fe57e2a4640 10 monclient(hunting): authenticate will time out at 23074.960793s 2024-03-28T22:16:47.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.819+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe57806f600 legacy=0x7fe57813faa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:16:47.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.819+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe57806f9e0 legacy=0x7fe57813c620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:16:47.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.819+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 msgr2=0x7fe57814ab30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:16:47.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:47.819+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 0x7fe57814ab30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:16:48.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:48.020+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe57806f600 legacy=0x7fe57813faa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:16:48.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:48.020+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe57806f9e0 legacy=0x7fe57813c620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:16:48.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:48.020+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 msgr2=0x7fe57814ab30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:16:48.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:48.020+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 0x7fe57814ab30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:16:48.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:48.421+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe57806f600 legacy=0x7fe57813faa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:16:48.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:48.421+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe57806f9e0 legacy=0x7fe57813c620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:16:48.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:48.421+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 msgr2=0x7fe57814ab30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:16:48.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:48.421+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 0x7fe57814ab30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:16:49.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:49.222+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe57806f600 legacy=0x7fe57813faa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:16:49.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:49.222+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 msgr2=0x7fe57814ab30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:16:49.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:49.222+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe57806f9e0 legacy=0x7fe57813c620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:16:49.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:49.222+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 0x7fe57814ab30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:16:50.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:16:50.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:16:50.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:16:50.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:16:50.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe575ffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 msgr2=0x7fe57814ab30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:16:50.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 0x7fe57814ab30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:50.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe575ffb640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe57806f9e0 legacy=0x7fe57813c620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:16:50.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe575ffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe57806f600 legacy=0x7fe57813faa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:16:50.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe575ffb640 10 monclient: _add_conns ranks=[2,3] 2024-03-28T22:16:50.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640036a0 0x7fe564005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:50.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe5640036a0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T22:16:50.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564006130 0x7fe5640085f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:50.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe564006130 addr v2:172.21.15.7:3300/0 2024-03-28T22:16:50.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:16:50.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe575ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564006130 2024-03-28T22:16:50.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:16:50.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe575ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe564008cb0 con 0x7fe5640036a0 2024-03-28T22:16:50.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:16:50.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640036a0 msgr2=0x7fe564005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:16:50.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564006130 msgr2=0x7fe5640085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:16:50.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640036a0 0x7fe564005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:16:50.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:50.819+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564006130 0x7fe5640085f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:16:51.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:51.020+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564006130 msgr2=0x7fe5640085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:16:51.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:51.020+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640036a0 msgr2=0x7fe564005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:16:51.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:51.020+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564006130 0x7fe5640085f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:16:51.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:51.020+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640036a0 0x7fe564005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:16:51.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:51.421+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564006130 msgr2=0x7fe5640085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:16:51.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:51.421+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640036a0 msgr2=0x7fe564005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:16:51.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:51.421+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564006130 0x7fe5640085f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:16:51.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:51.421+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640036a0 0x7fe564005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:16:52.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:52.222+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564006130 msgr2=0x7fe5640085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:16:52.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:52.222+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640036a0 msgr2=0x7fe564005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:16:52.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:52.222+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564006130 0x7fe5640085f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:16:52.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:52.222+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640036a0 0x7fe564005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:16:53.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:16:53.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:16:53.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:16:53.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:16:53.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe575ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640036a0 msgr2=0x7fe564005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:16:53.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640036a0 0x7fe564005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe575ffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564006130 msgr2=0x7fe5640085f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:16:53.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564006130 0x7fe5640085f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:53.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe575ffb640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-03-28T22:16:53.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 0x7fe56400af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe5780ad5d0 addr v2:172.21.15.83:3300/0 2024-03-28T22:16:53.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe57c841640 1 -- reap_dead start 2024-03-28T22:16:53.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe57806f9e0 addr v1:172.21.15.83:6789/0 2024-03-28T22:16:53.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe5640065e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:53.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe564003ae0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T22:16:53.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe575ffb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5781401b0 con 0x7fe57806f9e0 2024-03-28T22:16:53.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:16:53.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe575ffb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe5780ad5d0 2024-03-28T22:16:53.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:16:53.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe575ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe564008cb0 con 0x7fe564003ae0 2024-03-28T22:16:53.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:16:53.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe57c841640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe5640065e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:16:53.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe57c841640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe5640065e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:16:53.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe57806f9e0 legacy=0x7fe56400d4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:16:53.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 msgr2=0x7fe56400af60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:16:53.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:53.820+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 0x7fe56400af60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:16:54.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:54.021+0000 7fe57c841640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe5640065e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:16:54.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:54.021+0000 7fe57c841640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe5640065e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:16:54.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:54.021+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe57806f9e0 legacy=0x7fe56400d4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:16:54.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:54.021+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 msgr2=0x7fe56400af60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:16:54.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:54.021+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 0x7fe56400af60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:16:54.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:54.422+0000 7fe57c841640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe5640065e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:16:54.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:54.422+0000 7fe57c841640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe5640065e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:16:54.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:54.422+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe57806f9e0 legacy=0x7fe56400d4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:16:54.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:54.422+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 msgr2=0x7fe56400af60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:16:54.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:54.422+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 0x7fe56400af60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:16:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:55.223+0000 7fe57c841640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe5640065e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:16:55.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:55.223+0000 7fe57c841640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe5640065e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:16:55.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:55.223+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe57806f9e0 legacy=0x7fe56400d4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:16:55.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:55.223+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 msgr2=0x7fe56400af60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:16:55.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:55.223+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 0x7fe56400af60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:16:56.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:56.820+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:16:56.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:56.820+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:16:56.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:56.820+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:16:56.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:56.820+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:16:56.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:56.820+0000 7fe575ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe5640065e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:16:56.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:56.820+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe5640065e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:56.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:56.820+0000 7fe575ffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 msgr2=0x7fe56400af60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:16:56.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:56.820+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5780ad5d0 0x7fe56400af60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:56.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:56.820+0000 7fe575ffb640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe57806f9e0 legacy=0x7fe56400d4d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:16:56.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:56.820+0000 7fe575ffb640 10 monclient: _add_conns ranks=[3,0] 2024-03-28T22:16:56.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:56.820+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564007d20 0x7fe564008100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:56.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:56.820+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe564007d20 addr v2:172.21.15.7:3300/0 2024-03-28T22:16:56.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:56.820+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe564006bb0 addr v1:172.21.15.7:6789/0 2024-03-28T22:16:56.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:56.820+0000 7fe575ffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780acf90 con 0x7fe564006bb0 2024-03-28T22:16:56.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:56.820+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:16:56.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:56.820+0000 7fe575ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564007d20 2024-03-28T22:16:56.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:56.820+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:16:56.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:56.820+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564007d20 msgr2=0x7fe564008100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:16:56.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:56.820+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564006bb0 legacy=0x7fe564006fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:16:56.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:56.820+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564007d20 0x7fe564008100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:16:57.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:57.021+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564007d20 msgr2=0x7fe564008100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:16:57.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:57.021+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564006bb0 legacy=0x7fe564006fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:16:57.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:57.021+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564007d20 0x7fe564008100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:16:57.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:57.422+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564007d20 msgr2=0x7fe564008100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:16:57.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:57.422+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564006bb0 legacy=0x7fe564006fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:16:57.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:57.422+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564007d20 0x7fe564008100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:16:58.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:58.223+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564007d20 msgr2=0x7fe564008100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:16:58.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:58.223+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564006bb0 legacy=0x7fe564006fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:16:58.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:58.223+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564007d20 0x7fe564008100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:16:59.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.820+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:16:59.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.820+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:16:59.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.820+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:16:59.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.820+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:16:59.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.820+0000 7fe575ffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564007d20 msgr2=0x7fe564008100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:16:59.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.820+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564007d20 0x7fe564008100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:16:59.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.820+0000 7fe575ffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564006bb0 legacy=0x7fe564006fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:16:59.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.820+0000 7fe575ffb640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-03-28T22:16:59.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.820+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe5640065e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:59.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.820+0000 7fe57c841640 1 -- reap_dead start 2024-03-28T22:16:59.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.820+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe564003ae0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T22:16:59.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.820+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe564008100 addr v1:172.21.15.7:6789/0 2024-03-28T22:16:59.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.820+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe56400af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:16:59.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.820+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe5640076a0 addr v2:172.21.15.83:3300/0 2024-03-28T22:16:59.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.820+0000 7fe575ffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5781401b0 con 0x7fe564008100 2024-03-28T22:16:59.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.821+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe5640065e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:16:59.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.821+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:16:59.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.821+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:16:59.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.821+0000 7fe575ffb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe5640076a0 2024-03-28T22:16:59.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.821+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:16:59.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.821+0000 7fe575ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe564008cb0 con 0x7fe564003ae0 2024-03-28T22:16:59.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.821+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe5640065e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:16:59.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.821+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:16:59.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.821+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe56400af40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:16:59.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:16:59.821+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe56400af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:00.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:00.022+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe5640065e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:00.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:00.022+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:00.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:00.022+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe5640065e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:00.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:00.022+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe56400af40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:00.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:00.022+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe56400af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:00.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:00.423+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe5640065e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:00.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:00.423+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:00.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:00.423+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe5640065e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:00.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:00.423+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe56400af40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:00.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:00.423+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe56400af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:01.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:01.224+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe5640065e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:01.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:01.224+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:01.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:01.224+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe5640065e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:01.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:01.224+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe56400af40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:01.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:01.224+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe56400af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:02.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:02.821+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:17:02.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:02.821+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:17:02.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:02.821+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:17:02.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:02.821+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:17:02.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:02.821+0000 7fe575ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe5640065e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:02.821+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe5640065e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:02.821+0000 7fe575ffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe56400af40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:02.821+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe56400af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:02.821+0000 7fe575ffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:02.821+0000 7fe575ffb640 10 monclient: _add_conns ranks=[1,3] 2024-03-28T22:17:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:02.821+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe56400e4e0 addr v1:172.21.15.83:6789/0 2024-03-28T22:17:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:02.821+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564011ff0 0x7fe564014400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:02.821+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe564011ff0 addr v2:172.21.15.7:3300/0 2024-03-28T22:17:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:02.821+0000 7fe575ffb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780acf90 con 0x7fe56400e4e0 2024-03-28T22:17:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:02.821+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:02.821+0000 7fe575ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564011ff0 2024-03-28T22:17:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:02.821+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:17:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:02.821+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564011ff0 msgr2=0x7fe564014400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:02.821+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564011ff0 0x7fe564014400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:02.821+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe56400e4e0 legacy=0x7fe5640108d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:03.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:03.022+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564011ff0 msgr2=0x7fe564014400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:03.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:03.022+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564011ff0 0x7fe564014400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:03.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:03.022+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe56400e4e0 legacy=0x7fe5640108d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:03.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:03.423+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564011ff0 msgr2=0x7fe564014400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:03.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:03.423+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564011ff0 0x7fe564014400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:03.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:03.423+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe56400e4e0 legacy=0x7fe5640108d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:04.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:04.224+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564011ff0 msgr2=0x7fe564014400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:04.224+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564011ff0 0x7fe564014400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:04.224+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe56400e4e0 legacy=0x7fe5640108d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:05.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:17:05.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:17:05.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:17:05.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:17:05.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe575ffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564011ff0 msgr2=0x7fe564014400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:05.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564011ff0 0x7fe564014400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:05.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe575ffb640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe56400e4e0 legacy=0x7fe5640108d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:05.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe575ffb640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-03-28T22:17:05.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe57c841640 1 -- reap_dead start 2024-03-28T22:17:05.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe564008100 addr v1:172.21.15.7:6789/0 2024-03-28T22:17:05.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe56400af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:05.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe564003ae0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T22:17:05.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:05.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe5640076a0 addr v2:172.21.15.83:3300/0 2024-03-28T22:17:05.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:05.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe56400af40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:05.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe575ffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5781401b0 con 0x7fe564008100 2024-03-28T22:17:05.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe575ffb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe5640076a0 2024-03-28T22:17:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe575ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe564008cb0 con 0x7fe564003ae0 2024-03-28T22:17:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:17:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe56400af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:05.821+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:06.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:06.022+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:06.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:06.022+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe56400af40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:06.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:06.022+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe56400af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:06.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:06.022+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:06.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:06.022+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:06.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:06.423+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe56400af40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:06.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:06.423+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:06.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:06.423+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe56400af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:06.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:06.423+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:06.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:06.423+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:07.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:07.224+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:07.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:07.224+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe56400af40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:07.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:07.224+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe56400af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:07.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:07.224+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:07.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:07.224+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:08.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:08.821+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:17:08.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:08.821+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:17:08.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:08.821+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:17:08.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:08.821+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:17:08.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:08.821+0000 7fe575ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe56400af40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:08.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:08.821+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe56400af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:08.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:08.821+0000 7fe575ffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:08.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:08.821+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:08.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:08.821+0000 7fe575ffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:08.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:08.821+0000 7fe575ffb640 10 monclient: _add_conns ranks=[1,3] 2024-03-28T22:17:08.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:08.821+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe564014a50 addr v1:172.21.15.83:6789/0 2024-03-28T22:17:08.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:08.821+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 0x7fe564018b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:08.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:08.821+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe5640156d0 addr v2:172.21.15.7:3300/0 2024-03-28T22:17:08.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:08.821+0000 7fe575ffb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780acf90 con 0x7fe564014a50 2024-03-28T22:17:08.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:08.821+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:08.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:08.821+0000 7fe575ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe5640156d0 2024-03-28T22:17:08.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:08.821+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:17:08.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:08.821+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 msgr2=0x7fe564018b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:08.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:08.821+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 0x7fe564018b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:08.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:08.822+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564014a50 legacy=0x7fe564014e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:09.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:09.022+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 msgr2=0x7fe564018b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:09.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:09.022+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 0x7fe564018b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:09.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:09.023+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564014a50 legacy=0x7fe564014e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:09.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:09.423+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 msgr2=0x7fe564018b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:09.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:09.423+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 0x7fe564018b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:09.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:09.424+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564014a50 legacy=0x7fe564014e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:10.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:10.224+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 msgr2=0x7fe564018b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:10.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:10.224+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 0x7fe564018b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:10.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:10.225+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564014a50 legacy=0x7fe564014e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:11.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:17:11.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:17:11.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:17:11.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:17:11.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe575ffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 msgr2=0x7fe564018b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:11.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 0x7fe564018b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:11.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe575ffb640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564014a50 legacy=0x7fe564014e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:11.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe575ffb640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-03-28T22:17:11.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe57c841640 1 -- reap_dead start 2024-03-28T22:17:11.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe56400af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:11.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe564003ae0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T22:17:11.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564014a50 0x7fe564006510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:11.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe564014a50 addr v2:172.21.15.83:3300/0 2024-03-28T22:17:11.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 0x7fe564014e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:11.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe5640156d0 addr v2:172.21.15.7:3300/0 2024-03-28T22:17:11.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:11.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe56400af40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:11.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe575ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe5640156d0 2024-03-28T22:17:11.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:11.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe575ffb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe564008cb0 con 0x7fe564014a50 2024-03-28T22:17:11.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:11.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe575ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe564015370 con 0x7fe564003ae0 2024-03-28T22:17:11.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe56400af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:11.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe57c841640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 msgr2=0x7fe564014e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:11.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:17:11.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe57c841640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 0x7fe564014e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:11.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564014a50 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:11.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:11.822+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564014a50 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:12.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:12.023+0000 7fe57c841640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 msgr2=0x7fe564014e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:12.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:12.023+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe56400af40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:12.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:12.023+0000 7fe57c841640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 0x7fe564014e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:12.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:12.023+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe56400af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:12.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:12.023+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564014a50 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:12.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:12.023+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564014a50 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:12.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:12.424+0000 7fe57c841640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 msgr2=0x7fe564014e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:12.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:12.424+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe56400af40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:12.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:12.424+0000 7fe57c841640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 0x7fe564014e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:12.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:12.424+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe56400af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:12.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:12.424+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564014a50 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:12.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:12.424+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564014a50 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:13.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:13.225+0000 7fe57c841640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 msgr2=0x7fe564014e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:13.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:13.225+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe56400af40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:13.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:13.225+0000 7fe57c841640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 0x7fe564014e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:13.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:13.225+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe56400af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:13.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:13.225+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564014a50 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:13.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:13.225+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564014a50 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:14.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:14.822+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:17:14.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:14.822+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:17:14.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:14.822+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:17:14.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:14.822+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:17:14.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:14.822+0000 7fe575ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe56400af40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:14.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:14.822+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe56400af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:14.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:14.822+0000 7fe575ffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564014a50 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:14.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:14.822+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564014a50 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:14.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:14.822+0000 7fe575ffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 msgr2=0x7fe564014e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:14.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:14.822+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640156d0 0x7fe564014e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:14.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:14.822+0000 7fe575ffb640 10 monclient: _add_conns ranks=[0,1] 2024-03-28T22:17:14.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:14.822+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe564008100 addr v1:172.21.15.7:6789/0 2024-03-28T22:17:14.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:14.822+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5640076a0 addr v1:172.21.15.83:6789/0 2024-03-28T22:17:14.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:14.822+0000 7fe575ffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5781401b0 con 0x7fe564008100 2024-03-28T22:17:14.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:14.822+0000 7fe575ffb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780acf90 con 0x7fe5640076a0 2024-03-28T22:17:14.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:14.822+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:17:14.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:14.822+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:14.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:14.822+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe5640076a0 legacy=0x7fe564019120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:15.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:15.023+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:15.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:15.023+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe5640076a0 legacy=0x7fe564019120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:15.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:15.424+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:15.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:15.424+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe5640076a0 legacy=0x7fe564019120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:16.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:16.225+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:16.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:16.225+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe5640076a0 legacy=0x7fe564019120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:17.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:17.823+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:17:17.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:17.823+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:17:17.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:17.823+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:17:17.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:17.823+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:17:17.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:17.823+0000 7fe575ffb640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe5640076a0 legacy=0x7fe564019120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:17.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:17.823+0000 7fe575ffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:17.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:17.823+0000 7fe575ffb640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-03-28T22:17:17.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:17.823+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe56400af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:17.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:17.823+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe564003ae0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T22:17:17.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:17.823+0000 7fe57c841640 1 -- reap_dead start 2024-03-28T22:17:17.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:17.823+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5640076a0 addr v1:172.21.15.83:6789/0 2024-03-28T22:17:17.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:17.823+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe5640156d0 addr v1:172.21.15.7:6789/0 2024-03-28T22:17:17.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:17.823+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe56400af40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:17.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:17.823+0000 7fe575ffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780709c0 con 0x7fe5640156d0 2024-03-28T22:17:17.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:17.823+0000 7fe575ffb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5781401b0 con 0x7fe5640076a0 2024-03-28T22:17:17.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:17.823+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:17.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:17.823+0000 7fe575ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564003ae0 2024-03-28T22:17:17.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:17.823+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:17:17.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:17.823+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe56400af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:17.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:17.823+0000 7fe57c841640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe5640156d0 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:17.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:17.823+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe5640076a0 legacy=0x7fe564017b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:18.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:18.024+0000 7fe57c841640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe5640156d0 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:18.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:18.024+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe56400af40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:18.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:18.024+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe56400af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:18.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:18.024+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe5640076a0 legacy=0x7fe564017b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:18.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:18.425+0000 7fe57c841640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe5640156d0 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:18.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:18.425+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe56400af40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:18.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:18.425+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe56400af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:18.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:18.425+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe5640076a0 legacy=0x7fe564017b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:19.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:19.226+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe56400af40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:19.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:19.226+0000 7fe57c841640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe5640156d0 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:19.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:19.226+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe56400af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:19.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:19.226+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe5640076a0 legacy=0x7fe564017b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:20.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:17:20.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:17:20.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:17:20.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:17:20.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe575ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe56400af40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:20.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe56400af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:20.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe575ffb640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe5640076a0 legacy=0x7fe564017b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:20.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe575ffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe5640156d0 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:20.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe575ffb640 10 monclient: _add_conns ranks=[3,4] 2024-03-28T22:17:20.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:20.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe564008100 addr v2:172.21.15.7:3300/0 2024-03-28T22:17:20.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564018cd0 0x7fe564018210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:20.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe564018cd0 addr v2:172.21.15.83:3300/0 2024-03-28T22:17:20.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:20.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe575ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564008100 2024-03-28T22:17:20.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:20.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe575ffb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe564008cb0 con 0x7fe564018cd0 2024-03-28T22:17:20.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:17:20.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:20.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:20.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564018cd0 msgr2=0x7fe564018210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:20.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:20.823+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564018cd0 0x7fe564018210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:21.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:21.024+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:21.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:21.024+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:21.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:21.024+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564018cd0 msgr2=0x7fe564018210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:21.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:21.024+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564018cd0 0x7fe564018210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:21.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:21.425+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:21.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:21.425+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:21.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:21.425+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564018cd0 msgr2=0x7fe564018210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:21.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:21.425+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564018cd0 0x7fe564018210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:22.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:22.226+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:22.226+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:22.226+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564018cd0 msgr2=0x7fe564018210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:22.226+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564018cd0 0x7fe564018210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:23.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:17:23.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:17:23.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:17:23.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:17:23.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe575ffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564018cd0 msgr2=0x7fe564018210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:23.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564018cd0 0x7fe564018210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:23.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe575ffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:23.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:23.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe575ffb640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-03-28T22:17:23.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5640076a0 addr v1:172.21.15.83:6789/0 2024-03-28T22:17:23.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe57c841640 1 -- reap_dead start 2024-03-28T22:17:23.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe56400d490 0x7fe564018210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:23.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe56400d490 addr v2:172.21.15.83:3300/0 2024-03-28T22:17:23.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:23.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe564008100 addr v2:172.21.15.7:3300/0 2024-03-28T22:17:23.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe575ffb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780acf90 con 0x7fe5640076a0 2024-03-28T22:17:23.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:23.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe575ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564008100 2024-03-28T22:17:23.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:23.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe575ffb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe564008cb0 con 0x7fe56400d490 2024-03-28T22:17:23.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:17:23.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe57c841640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:23.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe57c841640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:23.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe56400d490 msgr2=0x7fe564018210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:23.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe5640076a0 legacy=0x7fe564017b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:23.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:23.823+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe56400d490 0x7fe564018210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:24.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:24.024+0000 7fe57c841640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:24.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:24.024+0000 7fe57c841640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:24.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:24.024+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe56400d490 msgr2=0x7fe564018210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:24.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:24.024+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe56400d490 0x7fe564018210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:24.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:24.024+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe5640076a0 legacy=0x7fe564017b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:24.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:24.425+0000 7fe57c841640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:24.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:24.425+0000 7fe57c841640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:24.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:24.425+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe56400d490 msgr2=0x7fe564018210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:24.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:24.425+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe5640076a0 legacy=0x7fe564017b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:24.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:24.425+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe56400d490 0x7fe564018210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:25.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:25.226+0000 7fe57c841640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:25.226+0000 7fe57c841640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:25.226+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe56400d490 msgr2=0x7fe564018210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:25.226+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe5640076a0 legacy=0x7fe564017b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:25.226+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe56400d490 0x7fe564018210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:26.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:26.823+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:17:26.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:26.823+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:17:26.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:26.823+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:17:26.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:26.823+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:17:26.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:26.823+0000 7fe575ffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe56400d490 msgr2=0x7fe564018210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:26.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:26.823+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe56400d490 0x7fe564018210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:26.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:26.823+0000 7fe575ffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:26.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:26.823+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:26.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:26.823+0000 7fe575ffb640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe5640076a0 legacy=0x7fe564017b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:26.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:26.823+0000 7fe575ffb640 10 monclient: _add_conns ranks=[0,2] 2024-03-28T22:17:26.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:26.823+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe5640156d0 addr v1:172.21.15.7:6789/0 2024-03-28T22:17:26.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:26.823+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe564014900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:26.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:26.823+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe564003ae0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T22:17:26.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:26.823+0000 7fe575ffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780709c0 con 0x7fe5640156d0 2024-03-28T22:17:26.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:26.823+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:26.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:26.823+0000 7fe575ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564003ae0 2024-03-28T22:17:26.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:26.823+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:17:26.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:26.823+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe5640156d0 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:26.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:26.823+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:26.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:26.823+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:27.024+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe5640156d0 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:27.024+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:27.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:27.024+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:27.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:27.425+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe5640156d0 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:27.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:27.425+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:27.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:27.425+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:28.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:28.226+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe5640156d0 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:28.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:28.226+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:28.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:28.226+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:29.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:17:29.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:17:29.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:17:29.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:17:29.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe575ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:29.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003ae0 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:29.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe575ffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe5640156d0 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:29.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe575ffb640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-03-28T22:17:29.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:29.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe57c841640 1 -- reap_dead start 2024-03-28T22:17:29.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe564008100 addr v2:172.21.15.7:3300/0 2024-03-28T22:17:29.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe564003ae0 addr v1:172.21.15.83:6789/0 2024-03-28T22:17:29.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 0x7fe56400df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:29.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe5640156d0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T22:17:29.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:29.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe575ffb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780acf90 con 0x7fe564003ae0 2024-03-28T22:17:29.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:29.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe575ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564008100 2024-03-28T22:17:29.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:29.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe57c841640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 msgr2=0x7fe56400df60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:29.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:29.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe57c841640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 0x7fe56400df60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:29.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe575ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe564008cb0 con 0x7fe5640156d0 2024-03-28T22:17:29.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:17:29.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:29.824+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564003ae0 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:30.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:30.025+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:30.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:30.025+0000 7fe57c841640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 msgr2=0x7fe56400df60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:30.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:30.025+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:30.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:30.025+0000 7fe57c841640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 0x7fe56400df60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:30.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:30.025+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564003ae0 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:30.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:30.425+0000 7fe57c841640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 msgr2=0x7fe56400df60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:30.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:30.425+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:30.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:30.425+0000 7fe57c841640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 0x7fe56400df60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:30.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:30.425+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:30.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:30.425+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564003ae0 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:31.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:31.226+0000 7fe57c841640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 msgr2=0x7fe56400df60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:31.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:31.226+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:31.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:31.226+0000 7fe57c841640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 0x7fe56400df60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:31.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:31.226+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:31.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:31.226+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564003ae0 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:32.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:32.823+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:17:32.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:32.823+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:17:32.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:32.823+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:17:32.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:32.823+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:17:32.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:32.823+0000 7fe575ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 msgr2=0x7fe56400df60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:32.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:32.823+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 0x7fe56400df60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:32.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:32.823+0000 7fe575ffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:32.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:32.823+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:32.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:32.823+0000 7fe575ffb640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564003ae0 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:32.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:32.823+0000 7fe575ffb640 10 monclient: _add_conns ranks=[4,0] 2024-03-28T22:17:32.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:32.823+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564014900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:32.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:32.823+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe5640076a0 addr v2:172.21.15.83:3300/0 2024-03-28T22:17:32.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:32.823+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe564014e40 addr v1:172.21.15.7:6789/0 2024-03-28T22:17:32.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:32.823+0000 7fe575ffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780709c0 con 0x7fe564014e40 2024-03-28T22:17:32.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:32.823+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:32.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:32.823+0000 7fe575ffb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe5640076a0 2024-03-28T22:17:32.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:32.823+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:17:32.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:32.823+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe564017ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:32.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:32.823+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:32.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:32.823+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:33.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:33.024+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe564017ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:33.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:33.024+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:33.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:33.024+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:33.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:33.425+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe564017ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:33.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:33.425+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:33.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:33.425+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:34.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:34.226+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe564017ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:34.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:34.226+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:34.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:34.226+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:35.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:17:35.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:17:35.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:17:35.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:17:35.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe575ffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:35.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:35.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe575ffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe564017ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:35.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe575ffb640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-03-28T22:17:35.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564014900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:35.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe5640076a0 addr v2:172.21.15.83:3300/0 2024-03-28T22:17:35.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:35.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe564008100 addr v2:172.21.15.7:3300/0 2024-03-28T22:17:35.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe57c841640 1 -- reap_dead start 2024-03-28T22:17:35.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe564014e40 addr v1:172.21.15.7:6789/0 2024-03-28T22:17:35.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe575ffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780acf90 con 0x7fe564014e40 2024-03-28T22:17:35.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:35.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe575ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564008100 2024-03-28T22:17:35.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:35.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe575ffb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe564008cb0 con 0x7fe5640076a0 2024-03-28T22:17:35.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:17:35.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:35.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:35.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe57c841640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe564017ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:35.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:35.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:35.823+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:36.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:36.024+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:36.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:36.024+0000 7fe57c841640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe564017ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:36.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:36.024+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:36.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:36.024+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:36.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:36.024+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:36.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:36.425+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:36.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:36.425+0000 7fe57c841640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe564017ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:36.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:36.425+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:36.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:36.425+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:36.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:36.425+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:37.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:37.226+0000 7fe57c841640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe564017ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:37.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:37.226+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:37.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:37.226+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:37.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:37.226+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:37.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:37.226+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:38.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:38.823+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:17:38.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:38.823+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:17:38.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:38.823+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:17:38.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:38.823+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:17:38.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:38.823+0000 7fe575ffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:38.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:38.823+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:38.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:38.823+0000 7fe575ffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:38.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:38.823+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:38.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:38.823+0000 7fe575ffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe564017ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:38.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:38.823+0000 7fe575ffb640 10 monclient: _add_conns ranks=[2,1] 2024-03-28T22:17:38.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:38.823+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 0x7fe5640181d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:38.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:38.823+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe5640156d0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T22:17:38.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:38.823+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe564003ae0 addr v1:172.21.15.83:6789/0 2024-03-28T22:17:38.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:38.823+0000 7fe575ffb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780709c0 con 0x7fe564003ae0 2024-03-28T22:17:38.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:38.823+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:38.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:38.823+0000 7fe575ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe5640156d0 2024-03-28T22:17:38.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:38.823+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:17:38.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:38.823+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 msgr2=0x7fe5640181d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:38.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:38.823+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 0x7fe5640181d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:38.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:38.823+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564003ae0 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:39.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:39.024+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 msgr2=0x7fe5640181d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:39.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:39.024+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 0x7fe5640181d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:39.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:39.024+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564003ae0 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:39.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:39.425+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 msgr2=0x7fe5640181d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:39.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:39.425+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 0x7fe5640181d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:39.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:39.425+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564003ae0 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:40.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:40.226+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 msgr2=0x7fe5640181d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:40.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:40.226+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 0x7fe5640181d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:40.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:40.226+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564003ae0 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:41.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:17:41.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:17:41.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:17:41.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:17:41.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe575ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 msgr2=0x7fe5640181d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:41.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe5640156d0 0x7fe5640181d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:41.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe575ffb640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564003ae0 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:41.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe575ffb640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-03-28T22:17:41.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564014900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:41.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe5640076a0 addr v2:172.21.15.83:3300/0 2024-03-28T22:17:41.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe57c841640 1 -- reap_dead start 2024-03-28T22:17:41.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:41.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe564008100 addr v2:172.21.15.7:3300/0 2024-03-28T22:17:41.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe564014e40 addr v1:172.21.15.7:6789/0 2024-03-28T22:17:41.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe575ffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780acf90 con 0x7fe564014e40 2024-03-28T22:17:41.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:41.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe575ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564008100 2024-03-28T22:17:41.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:41.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe575ffb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe564008cb0 con 0x7fe5640076a0 2024-03-28T22:17:41.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:17:41.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:41.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:41.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe57c841640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:41.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:41.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:41.824+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:42.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:42.025+0000 7fe57c841640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:42.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:42.025+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:42.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:42.025+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:42.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:42.025+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:42.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:42.025+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:42.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:42.426+0000 7fe57c841640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:42.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:42.426+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:42.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:42.426+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:42.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:42.426+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:42.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:42.426+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:43.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:43.227+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:43.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:43.227+0000 7fe57c841640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:43.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:43.227+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:43.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:43.227+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:43.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:43.227+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:44.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:44.824+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:17:44.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:44.824+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:17:44.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:44.824+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:17:44.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:44.824+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:17:44.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:44.824+0000 7fe575ffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:44.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:44.824+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:44.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:44.824+0000 7fe575ffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:44.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:44.824+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:44.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:44.824+0000 7fe575ffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:44.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:44.824+0000 7fe575ffb640 10 monclient: _add_conns ranks=[1,2] 2024-03-28T22:17:44.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:44.824+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe564003a80 addr v1:172.21.15.83:6789/0 2024-03-28T22:17:44.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:44.824+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe56400df60 0x7fe5640197f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:44.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:44.824+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe56400df60 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T22:17:44.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:44.824+0000 7fe575ffb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780709c0 con 0x7fe564003a80 2024-03-28T22:17:44.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:44.824+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:44.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:44.824+0000 7fe575ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe56400df60 2024-03-28T22:17:44.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:44.824+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:17:44.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:44.824+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe56400df60 msgr2=0x7fe5640197f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:44.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:44.824+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe56400df60 0x7fe5640197f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:44.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:44.824+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564003a80 legacy=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:45.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:45.025+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe56400df60 msgr2=0x7fe5640197f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:45.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:45.025+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe56400df60 0x7fe5640197f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:45.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:45.025+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564003a80 legacy=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:45.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:45.426+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe56400df60 msgr2=0x7fe5640197f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:45.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:45.426+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe56400df60 0x7fe5640197f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:45.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:45.426+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564003a80 legacy=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:46.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:46.227+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe56400df60 msgr2=0x7fe5640197f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:46.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:46.227+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe56400df60 0x7fe5640197f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:46.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:46.227+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564003a80 legacy=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:47.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:17:47.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:17:47.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:17:47.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:17:47.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe575ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe56400df60 msgr2=0x7fe5640197f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:47.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe56400df60 0x7fe5640197f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:47.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe575ffb640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564003a80 legacy=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:47.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe575ffb640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-03-28T22:17:47.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe564014e40 addr v1:172.21.15.7:6789/0 2024-03-28T22:17:47.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe57c841640 1 -- reap_dead start 2024-03-28T22:17:47.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 0x7fe56401a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:47.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe564019d30 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T22:17:47.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 0x7fe564014900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:47.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:47.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe564003a80 addr v2:172.21.15.83:3300/0 2024-03-28T22:17:47.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 msgr2=0x7fe56401a2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:47.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe575ffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780acf90 con 0x7fe564014e40 2024-03-28T22:17:47.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:47.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe575ffb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564003a80 2024-03-28T22:17:47.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:47.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe575ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe564008cb0 con 0x7fe564019d30 2024-03-28T22:17:47.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:17:47.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 0x7fe56401a2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:47.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:47.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:47.825+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:48.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:48.026+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:48.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:48.026+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 msgr2=0x7fe56401a2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:48.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:48.026+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 0x7fe56401a2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:48.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:48.026+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:48.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:48.026+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:48.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:48.427+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:48.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:48.427+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 msgr2=0x7fe56401a2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:48.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:48.427+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 0x7fe56401a2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:48.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:48.427+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:48.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:48.427+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:49.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:49.228+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:49.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:49.228+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 msgr2=0x7fe56401a2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:49.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:49.228+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 0x7fe56401a2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:49.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:49.228+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:49.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:49.228+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:50.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:50.825+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:17:50.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:50.825+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:17:50.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:50.825+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:17:50.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:50.825+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:17:50.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:50.825+0000 7fe575ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 msgr2=0x7fe56401a2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:50.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:50.825+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 0x7fe56401a2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:50.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:50.825+0000 7fe575ffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:50.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:50.825+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:50.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:50.825+0000 7fe575ffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:50.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:50.825+0000 7fe575ffb640 10 monclient: _add_conns ranks=[3,1] 2024-03-28T22:17:50.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:50.825+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:50.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:50.825+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe5640076a0 addr v2:172.21.15.7:3300/0 2024-03-28T22:17:50.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:50.825+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe564008100 addr v1:172.21.15.83:6789/0 2024-03-28T22:17:50.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:50.825+0000 7fe575ffb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780709c0 con 0x7fe564008100 2024-03-28T22:17:50.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:50.825+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:50.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:50.825+0000 7fe575ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe5640076a0 2024-03-28T22:17:50.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:50.825+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:17:50.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:50.825+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:50.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:50.825+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:50.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:50.825+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564008100 legacy=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:51.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:51.026+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:51.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:51.026+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:51.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:51.026+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564008100 legacy=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:51.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:51.427+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:51.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:51.427+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:51.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:51.427+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564008100 legacy=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:52.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:52.228+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:52.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:52.228+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:52.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:52.228+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564008100 legacy=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:53.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:17:53.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:17:53.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:17:53.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:17:53.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe575ffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:53.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:53.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe575ffb640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564008100 legacy=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:53.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe575ffb640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-03-28T22:17:53.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 0x7fe564014900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:53.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe564003a80 addr v2:172.21.15.83:3300/0 2024-03-28T22:17:53.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 0x7fe5640197f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe564019d30 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T22:17:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe57c841640 1 -- reap_dead start 2024-03-28T22:17:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe5640076a0 addr v2:172.21.15.7:3300/0 2024-03-28T22:17:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe575ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe5640076a0 2024-03-28T22:17:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe575ffb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe564008cb0 con 0x7fe564003a80 2024-03-28T22:17:53.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe575ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe564015370 con 0x7fe564019d30 2024-03-28T22:17:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:17:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 msgr2=0x7fe5640197f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe57c841640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 0x7fe5640197f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe57c841640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:53.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:53.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:53.824+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:54.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:54.026+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 msgr2=0x7fe5640197f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:54.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:54.026+0000 7fe57c841640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:54.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:54.026+0000 7fe57c841640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:54.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:54.026+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 0x7fe5640197f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:54.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:54.026+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:54.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:54.026+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:17:54.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:54.427+0000 7fe57c841640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:54.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:54.427+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 msgr2=0x7fe5640197f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:54.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:54.427+0000 7fe57c841640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:54.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:54.427+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 0x7fe5640197f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:54.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:54.427+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:54.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:54.427+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:17:55.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:55.228+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 msgr2=0x7fe5640197f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:55.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:55.228+0000 7fe57c841640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:55.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:55.228+0000 7fe57c841640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:55.228+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 0x7fe5640197f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:55.228+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:17:55.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:55.228+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:17:56.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:56.825+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:17:56.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:56.825+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:17:56.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:56.825+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:17:56.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:56.825+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:17:56.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:56.825+0000 7fe575ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 msgr2=0x7fe5640197f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:56.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:56.825+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564019d30 0x7fe5640197f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:56.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:56.825+0000 7fe575ffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:56.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:56.825+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564003a80 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:56.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:56.825+0000 7fe575ffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:56.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:56.825+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:17:56.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:56.825+0000 7fe575ffb640 10 monclient: _add_conns ranks=[0,1] 2024-03-28T22:17:56.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:56.825+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe564014e40 addr v1:172.21.15.7:6789/0 2024-03-28T22:17:56.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:56.825+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe564008100 addr v1:172.21.15.83:6789/0 2024-03-28T22:17:56.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:56.825+0000 7fe575ffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780acf90 con 0x7fe564014e40 2024-03-28T22:17:56.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:56.825+0000 7fe575ffb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780709c0 con 0x7fe564008100 2024-03-28T22:17:56.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:56.825+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:17:56.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:56.826+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:56.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:56.826+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:57.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:57.027+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:57.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:57.027+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:57.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:57.428+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:57.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:57.428+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:58.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:58.229+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:58.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:58.229+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:17:59.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:17:59.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:17:59.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:17:59.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:17:59.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe575ffb640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:59.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe575ffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564014e40 legacy=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:17:59.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe575ffb640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-03-28T22:17:59.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe57c841640 1 -- reap_dead start 2024-03-28T22:17:59.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe5640076a0 addr v2:172.21.15.7:3300/0 2024-03-28T22:17:59.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:17:59.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe564003a80 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T22:17:59.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe564008100 addr v1:172.21.15.7:6789/0 2024-03-28T22:17:59.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe575ffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5781401b0 con 0x7fe564008100 2024-03-28T22:17:59.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:17:59.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:59.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe575ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe5640076a0 2024-03-28T22:17:59.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:17:59.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:17:59.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe575ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe564008cb0 con 0x7fe564003a80 2024-03-28T22:17:59.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:17:59.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:17:59.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe57c841640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:17:59.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:17:59.826+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:00.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:00.027+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:00.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:00.027+0000 7fe57c841640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:00.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:00.027+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:00.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:00.027+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:00.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:00.027+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:00.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:00.428+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:00.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:00.428+0000 7fe57c841640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:00.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:00.428+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:00.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:00.428+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:00.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:00.428+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:01.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:01.229+0000 7fe57c841640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:01.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:01.229+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:01.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:01.229+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:01.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:01.229+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:01.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:01.229+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:02.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:02.826+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:18:02.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:02.826+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:18:02.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:02.826+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:18:02.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:02.826+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:18:02.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:02.826+0000 7fe575ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:02.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:02.826+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:02.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:02.826+0000 7fe575ffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:02.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:02.826+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:02.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:02.826+0000 7fe575ffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:02.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:02.826+0000 7fe575ffb640 10 monclient: _add_conns ranks=[4,1] 2024-03-28T22:18:02.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:02.826+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564009640 0x7fe564014900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:02.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:02.826+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe564009640 addr v2:172.21.15.83:3300/0 2024-03-28T22:18:02.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:02.826+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe564009a50 addr v1:172.21.15.83:6789/0 2024-03-28T22:18:02.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:02.826+0000 7fe575ffb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780acf90 con 0x7fe564009a50 2024-03-28T22:18:02.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:02.826+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:02.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:02.826+0000 7fe575ffb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564009640 2024-03-28T22:18:02.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:02.826+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:18:02.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:02.826+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009a50 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:02.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:02.826+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564009640 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:02.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:02.826+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564009640 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:03.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:03.027+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009a50 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:03.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:03.027+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564009640 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:03.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:03.027+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564009640 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:03.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:03.428+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009a50 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:03.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:03.428+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564009640 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:03.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:03.428+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564009640 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:04.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:04.229+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564009640 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:04.229+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009a50 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:04.229+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564009640 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:05.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:18:05.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:18:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:18:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:18:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe575ffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564009640 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564009640 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe575ffb640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009a50 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe575ffb640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-03-28T22:18:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe57c841640 1 -- reap_dead start 2024-03-28T22:18:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe5640076a0 addr v2:172.21.15.7:3300/0 2024-03-28T22:18:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe564003a80 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T22:18:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564009a50 0x7fe564008f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe564009a50 addr v2:172.21.15.83:3300/0 2024-03-28T22:18:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe575ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe5640076a0 2024-03-28T22:18:05.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:05.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:05.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:05.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:05.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe575ffb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe564008cb0 con 0x7fe564009a50 2024-03-28T22:18:05.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:05.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe575ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe564015370 con 0x7fe564003a80 2024-03-28T22:18:05.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:05.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.826+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:18:05.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.827+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564009a50 msgr2=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:05.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:05.827+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564009a50 0x7fe564008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:06.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:06.027+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:06.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:06.027+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:06.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:06.027+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:06.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:06.027+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:06.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:06.028+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564009a50 msgr2=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:06.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:06.028+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564009a50 0x7fe564008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:06.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:06.428+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:06.428+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:06.428+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:06.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:06.428+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:06.429+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564009a50 msgr2=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:06.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:06.429+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564009a50 0x7fe564008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:07.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:07.229+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:07.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:07.229+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:07.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:07.229+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:07.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:07.229+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:07.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:07.230+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564009a50 msgr2=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:07.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:07.230+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564009a50 0x7fe564008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:08.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:08.827+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:18:08.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:08.827+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:18:08.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:08.827+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:18:08.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:08.827+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:18:08.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:08.827+0000 7fe575ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:08.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:08.827+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:08.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:08.827+0000 7fe575ffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564009a50 msgr2=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:08.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:08.827+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564009a50 0x7fe564008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:08.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:08.827+0000 7fe575ffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:08.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:08.827+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe5640076a0 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:08.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:08.827+0000 7fe575ffb640 10 monclient: _add_conns ranks=[0,1] 2024-03-28T22:18:08.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:08.827+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe564008100 addr v1:172.21.15.7:6789/0 2024-03-28T22:18:08.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:08.827+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe564009470 addr v1:172.21.15.83:6789/0 2024-03-28T22:18:08.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:08.827+0000 7fe575ffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5781401b0 con 0x7fe564008100 2024-03-28T22:18:08.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:08.827+0000 7fe575ffb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780acf90 con 0x7fe564009470 2024-03-28T22:18:08.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:08.827+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:18:08.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:08.827+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:08.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:08.827+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009470 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:09.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:09.028+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:09.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:09.028+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009470 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:09.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:09.429+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:09.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:09.429+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009470 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:10.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:10.230+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:10.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:10.230+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009470 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:11.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:18:11.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:18:11.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:18:11.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:18:11.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe575ffb640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009470 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:11.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe575ffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564014900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:11.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe575ffb640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-03-28T22:18:11.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe57c841640 1 -- reap_dead start 2024-03-28T22:18:11.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe564009470 addr v1:172.21.15.83:6789/0 2024-03-28T22:18:11.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:11.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe564003a80 addr v2:172.21.15.7:3300/0 2024-03-28T22:18:11.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564008100 0x7fe564008f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:11.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe564008100 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T22:18:11.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe575ffb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780709c0 con 0x7fe564009470 2024-03-28T22:18:11.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:11.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564003a80 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:11.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe575ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564003a80 2024-03-28T22:18:11.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe57c841640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564008100 msgr2=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:11.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:11.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:11.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe575ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe564008cb0 con 0x7fe564008100 2024-03-28T22:18:11.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe57c841640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564008100 0x7fe564008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:11.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:18:11.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:11.827+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009470 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:12.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:12.028+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564003a80 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:12.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:12.028+0000 7fe57c841640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564008100 msgr2=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:12.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:12.028+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:12.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:12.028+0000 7fe57c841640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564008100 0x7fe564008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:12.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:12.028+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009470 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:12.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:12.429+0000 7fe57c841640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564008100 msgr2=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:12.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:12.429+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564003a80 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:12.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:12.429+0000 7fe57c841640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564008100 0x7fe564008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:12.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:12.429+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:12.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:12.429+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009470 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:13.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:13.230+0000 7fe5777fe640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564003a80 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:13.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:13.230+0000 7fe57c841640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564008100 msgr2=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:13.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:13.230+0000 7fe5777fe640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:13.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:13.230+0000 7fe57c841640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564008100 0x7fe564008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:13.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:13.230+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009470 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:14.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:14.828+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:18:14.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:14.828+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:18:14.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:14.828+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:18:14.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:14.828+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:18:14.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:14.828+0000 7fe575ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564008100 msgr2=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:14.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:14.828+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564008100 0x7fe564008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:14.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:14.828+0000 7fe575ffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564003a80 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:14.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:14.828+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:14.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:14.828+0000 7fe575ffb640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009470 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:14.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:14.828+0000 7fe575ffb640 10 monclient: _add_conns ranks=[4,0] 2024-03-28T22:18:14.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:14.828+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564009880 0x7fe564017df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:14.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:14.828+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe564009880 addr v2:172.21.15.83:3300/0 2024-03-28T22:18:14.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:14.828+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe564018330 addr v1:172.21.15.7:6789/0 2024-03-28T22:18:14.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:14.828+0000 7fe575ffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5781401b0 con 0x7fe564018330 2024-03-28T22:18:14.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:14.828+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:14.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:14.828+0000 7fe575ffb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564009880 2024-03-28T22:18:14.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:14.828+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:18:14.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:14.828+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564018330 legacy=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:14.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:14.828+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564009880 msgr2=0x7fe564017df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:14.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:14.828+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564009880 0x7fe564017df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:15.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:15.029+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564018330 legacy=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:15.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:15.029+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564009880 msgr2=0x7fe564017df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:15.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:15.029+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564009880 0x7fe564017df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:15.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:15.430+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564018330 legacy=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:15.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:15.430+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564009880 msgr2=0x7fe564017df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:15.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:15.430+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564009880 0x7fe564017df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:16.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:16.231+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564018330 legacy=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:16.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:16.231+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564009880 msgr2=0x7fe564017df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:16.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:16.231+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564009880 0x7fe564017df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:17.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:18:17.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:18:17.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:18:17.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:18:17.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe575ffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564009880 msgr2=0x7fe564017df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:17.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564009880 0x7fe564017df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:17.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe575ffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564018330 legacy=0x7fe564014900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:17.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe575ffb640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-03-28T22:18:17.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe57c841640 1 -- reap_dead start 2024-03-28T22:18:17.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe564009470 addr v1:172.21.15.83:6789/0 2024-03-28T22:18:17.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:17.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe564003a80 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T22:18:17.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 0x7fe564008f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:17.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe564008100 addr v2:172.21.15.83:3300/0 2024-03-28T22:18:17.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe575ffb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780709c0 con 0x7fe564009470 2024-03-28T22:18:17.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:17.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe575ffb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564008100 2024-03-28T22:18:17.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:17.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe575ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe564008cb0 con 0x7fe564003a80 2024-03-28T22:18:17.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:18:17.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:17.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009470 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:17.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:17.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 msgr2=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:17.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:17.828+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 0x7fe564008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:18.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:18.029+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:18.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:18.029+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:18.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:18.029+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 msgr2=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:18.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:18.029+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009470 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:18.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:18.029+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 0x7fe564008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:18.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:18.430+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:18.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:18.430+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:18.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:18.430+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 msgr2=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:18.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:18.430+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009470 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:18.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:18.430+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 0x7fe564008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:19.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:19.231+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:19.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:19.231+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:19.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:19.231+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009470 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:19.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:19.231+0000 7fe57c841640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 msgr2=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:19.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:19.231+0000 7fe57c841640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 0x7fe564008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:20.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:20.828+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:18:20.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:20.828+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:18:20.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:20.828+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:18:20.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:20.828+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:18:20.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:20.828+0000 7fe575ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:20.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:20.828+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:20.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:20.828+0000 7fe575ffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 msgr2=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:20.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:20.828+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 0x7fe564008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:20.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:20.828+0000 7fe575ffb640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009470 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:20.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:20.828+0000 7fe575ffb640 10 monclient: _add_conns ranks=[3,0] 2024-03-28T22:18:20.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:20.828+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 0x7fe564017e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:20.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:20.828+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe564009880 addr v2:172.21.15.7:3300/0 2024-03-28T22:18:20.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:20.828+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe564018350 addr v1:172.21.15.7:6789/0 2024-03-28T22:18:20.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:20.828+0000 7fe575ffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5781401b0 con 0x7fe564018350 2024-03-28T22:18:20.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:20.828+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:20.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:20.828+0000 7fe575ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564009880 2024-03-28T22:18:20.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:20.828+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:18:20.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:20.828+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 msgr2=0x7fe564017e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:20.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:20.828+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564018350 legacy=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:20.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:20.828+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 0x7fe564017e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:21.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:21.029+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 msgr2=0x7fe564017e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:21.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:21.029+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564018350 legacy=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:21.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:21.029+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 0x7fe564017e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:21.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:21.430+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 msgr2=0x7fe564017e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:21.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:21.430+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564018350 legacy=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:21.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:21.430+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 0x7fe564017e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:22.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:22.231+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564018350 legacy=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:22.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:22.231+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 msgr2=0x7fe564017e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:22.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:22.231+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 0x7fe564017e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:23.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:18:23.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:18:23.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:18:23.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:18:23.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe575ffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 msgr2=0x7fe564017e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:23.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 0x7fe564017e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:23.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe575ffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564018350 legacy=0x7fe564014900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:23.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe575ffb640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-03-28T22:18:23.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 0x7fe564017df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:23.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe564009880 addr v2:172.21.15.7:3300/0 2024-03-28T22:18:23.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe564009470 addr v1:172.21.15.83:6789/0 2024-03-28T22:18:23.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe57c841640 1 -- reap_dead start 2024-03-28T22:18:23.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:23.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe564003a80 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T22:18:23.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe575ffb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780709c0 con 0x7fe564009470 2024-03-28T22:18:23.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:23.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe575ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564009880 2024-03-28T22:18:23.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:23.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe575ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe564008cb0 con 0x7fe564003a80 2024-03-28T22:18:23.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:18:23.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 msgr2=0x7fe564017df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:23.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 0x7fe564017df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:23.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe57c841640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:23.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe57c841640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:23.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:23.828+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009470 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:24.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:24.029+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 msgr2=0x7fe564017df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:24.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:24.029+0000 7fe57c841640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:24.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:24.029+0000 7fe57c841640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:24.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:24.029+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 0x7fe564017df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:24.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:24.029+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009470 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:24.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:24.430+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 msgr2=0x7fe564017df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:24.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:24.430+0000 7fe57c841640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:24.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:24.430+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 0x7fe564017df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:24.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:24.430+0000 7fe57c841640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:24.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:24.430+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009470 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:25.231+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 msgr2=0x7fe564017df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:25.231+0000 7fe57c841640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:25.231+0000 7fe57c841640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:25.231+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 0x7fe564017df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:25.231+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009470 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:26.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:26.829+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:18:26.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:26.829+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:18:26.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:26.829+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:18:26.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:26.829+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:18:26.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:26.829+0000 7fe575ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:26.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:26.829+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:26.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:26.829+0000 7fe575ffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 msgr2=0x7fe564017df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:26.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:26.829+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 0x7fe564017df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:26.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:26.829+0000 7fe575ffb640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564009470 legacy=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:26.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:26.829+0000 7fe575ffb640 10 monclient: _add_conns ranks=[0,4] 2024-03-28T22:18:26.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:26.829+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe564008100 addr v1:172.21.15.7:6789/0 2024-03-28T22:18:26.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:26.829+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564008f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:26.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:26.829+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe5640076a0 addr v2:172.21.15.83:3300/0 2024-03-28T22:18:26.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:26.829+0000 7fe575ffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5781401b0 con 0x7fe564008100 2024-03-28T22:18:26.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:26.829+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:26.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:26.829+0000 7fe575ffb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe5640076a0 2024-03-28T22:18:26.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:26.829+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:18:26.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:26.829+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:26.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:26.829+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:26.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:26.829+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:27.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:27.030+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:27.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:27.030+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:27.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:27.030+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:27.431+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:27.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:27.431+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:27.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:27.431+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:28.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:28.232+0000 7fe577fff640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:28.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:28.232+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:28.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:28.232+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:29.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.828+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:18:29.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:18:29.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:18:29.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:18:29.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe575ffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 msgr2=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:29.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe5640076a0 0x7fe564008f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:29.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe575ffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564006f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:29.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe575ffb640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-03-28T22:18:29.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 0x7fe564017df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe57c841640 1 -- reap_dead start 2024-03-28T22:18:29.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe564009880 addr v2:172.21.15.7:3300/0 2024-03-28T22:18:29.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:29.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe564003a80 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T22:18:29.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe564008100 addr v1:172.21.15.7:6789/0 2024-03-28T22:18:29.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 msgr2=0x7fe564017df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:29.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe575ffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780709c0 con 0x7fe564008100 2024-03-28T22:18:29.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:29.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe575ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564009880 2024-03-28T22:18:29.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 0x7fe564017df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:29.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:29.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe575ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe564008cb0 con 0x7fe564003a80 2024-03-28T22:18:29.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:18:29.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:29.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe57c841640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:29.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:29.829+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:30.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:30.030+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 msgr2=0x7fe564017df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:30.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:30.030+0000 7fe57c841640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:30.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:30.030+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:30.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:30.030+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 0x7fe564017df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:30.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:30.030+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:30.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:30.431+0000 7fe57c841640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:30.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:30.431+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:30.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:30.431+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 msgr2=0x7fe564017df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:30.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:30.431+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:30.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:30.431+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 0x7fe564017df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:31.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:31.232+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 msgr2=0x7fe564017df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:31.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:31.232+0000 7fe57c841640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:31.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:31.232+0000 7fe5777fe640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:31.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:31.232+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 0x7fe564017df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:31.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:31.232+0000 7fe5777fe640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:32.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:32.828+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:18:32.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:32.828+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:18:32.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:32.828+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:18:32.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:32.828+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:18:32.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:32.828+0000 7fe575ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:32.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:32.828+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564003a80 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:32.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:32.828+0000 7fe575ffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 msgr2=0x7fe564017df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:32.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:32.828+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564009880 0x7fe564017df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:32.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:32.828+0000 7fe575ffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564008100 legacy=0x7fe564008f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:32.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:32.828+0000 7fe575ffb640 10 monclient: _add_conns ranks=[4,1] 2024-03-28T22:18:32.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:32.828+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564018330 0x7fe5640176e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:32.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:32.828+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe564018330 addr v2:172.21.15.83:3300/0 2024-03-28T22:18:32.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:32.828+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe564006f90 addr v1:172.21.15.83:6789/0 2024-03-28T22:18:32.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:32.828+0000 7fe575ffb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5781401b0 con 0x7fe564006f90 2024-03-28T22:18:32.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:32.828+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:32.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:32.828+0000 7fe575ffb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564018330 2024-03-28T22:18:32.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:32.828+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:18:32.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:32.828+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564018330 msgr2=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:32.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:32.828+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564006f90 legacy=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:32.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:32.828+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564018330 0x7fe5640176e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:33.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:33.029+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564018330 msgr2=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:33.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:33.029+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564006f90 legacy=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:33.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:33.029+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564018330 0x7fe5640176e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:33.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:33.430+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564018330 msgr2=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:33.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:33.430+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564006f90 legacy=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:33.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:33.430+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564018330 0x7fe5640176e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:34.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:34.231+0000 7fe577fff640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564018330 msgr2=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:34.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:34.231+0000 7fe5777fe640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564006f90 legacy=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:34.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:34.231+0000 7fe577fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564018330 0x7fe5640176e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:35.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:18:35.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:18:35.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:18:35.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:18:35.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe575ffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564018330 msgr2=0x7fe5640176e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:35.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564018330 0x7fe5640176e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:35.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe575ffb640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564006f90 legacy=0x7fe564014900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:35.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe575ffb640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-03-28T22:18:35.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe57c841640 1 -- reap_dead start 2024-03-28T22:18:35.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5640073a0 addr v1:172.21.15.83:6789/0 2024-03-28T22:18:35.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe564008100 addr v2:172.21.15.83:3300/0 2024-03-28T22:18:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 0x7fe564014900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe564005a90 addr v2:172.21.15.7:3300/0 2024-03-28T22:18:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe575ffb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780709c0 con 0x7fe5640073a0 2024-03-28T22:18:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe575ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564005a90 2024-03-28T22:18:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe575ffb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe564008cb0 con 0x7fe564008100 2024-03-28T22:18:35.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:18:35.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe57c841640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:35.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe57c841640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:35.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:35.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe5640073a0 legacy=0x7fe56401d310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:35.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:35.829+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:36.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:36.030+0000 7fe57c841640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:36.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:36.030+0000 7fe57c841640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:36.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:36.030+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe5640073a0 legacy=0x7fe56401d310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:36.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:36.030+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:36.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:36.030+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:36.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:36.431+0000 7fe57c841640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:36.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:36.431+0000 7fe57c841640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:36.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:36.431+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:36.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:36.431+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe5640073a0 legacy=0x7fe56401d310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:36.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:36.431+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:37.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:37.232+0000 7fe57c841640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:37.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:37.232+0000 7fe57c841640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:37.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:37.232+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe5640073a0 legacy=0x7fe56401d310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:37.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:37.232+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:37.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:37.232+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:38.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:38.828+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:18:38.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:38.828+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:18:38.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:38.828+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:18:38.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:38.828+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:18:38.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:38.828+0000 7fe575ffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:38.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:38.828+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:38.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:38.828+0000 7fe575ffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:38.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:38.828+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:38.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:38.828+0000 7fe575ffb640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe5640073a0 legacy=0x7fe56401d310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:38.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:38.828+0000 7fe575ffb640 10 monclient: _add_conns ranks=[2,0] 2024-03-28T22:18:38.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:38.828+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 0x7fe5640190e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:38.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:38.828+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe564014e40 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T22:18:38.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:38.829+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe564019620 addr v1:172.21.15.7:6789/0 2024-03-28T22:18:38.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:38.829+0000 7fe575ffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5781401b0 con 0x7fe564019620 2024-03-28T22:18:38.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:38.829+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:38.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:38.829+0000 7fe575ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564014e40 2024-03-28T22:18:38.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:38.829+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:18:38.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:38.829+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 msgr2=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:38.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:38.829+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564019620 legacy=0x7fe56401a2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:38.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:38.829+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 0x7fe5640190e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:39.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:39.030+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 msgr2=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:39.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:39.030+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564019620 legacy=0x7fe56401a2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:39.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:39.030+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 0x7fe5640190e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:39.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:39.431+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 msgr2=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:39.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:39.431+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564019620 legacy=0x7fe56401a2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:39.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:39.431+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 0x7fe5640190e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:40.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:40.232+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564019620 legacy=0x7fe56401a2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:40.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:40.232+0000 7fe577fff640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 msgr2=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:40.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:40.232+0000 7fe577fff640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 0x7fe5640190e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:41.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:18:41.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:18:41.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:18:41.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:18:41.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe575ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 msgr2=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:41.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 0x7fe5640190e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:41.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe575ffb640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564019620 legacy=0x7fe56401a2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:41.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe575ffb640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-03-28T22:18:41.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 0x7fe564014900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:41.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe564005a90 addr v2:172.21.15.7:3300/0 2024-03-28T22:18:41.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:41.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fe564008100 addr v2:172.21.15.83:3300/0 2024-03-28T22:18:41.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 0x7fe5640190e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T22:18:41.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe57c841640 1 -- reap_dead start 2024-03-28T22:18:41.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fe564014e40 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-03-28T22:18:41.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:41.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe575ffb640 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5780ada20 con 0x7fe564005a90 2024-03-28T22:18:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe575ffb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe564008cb0 con 0x7fe564008100 2024-03-28T22:18:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe575ffb640 10 monclient(hunting): start opening mon connection 2024-03-28T22:18:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe575ffb640 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe564015370 con 0x7fe564014e40 2024-03-28T22:18:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:18:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:41.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe57c841640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 msgr2=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:41.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:41.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:41.829+0000 7fe57c841640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 0x7fe5640190e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T22:18:42.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:42.030+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:42.030+0000 7fe57c841640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 msgr2=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:42.030+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:42.030+0000 7fe57c841640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 0x7fe5640190e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:42.030+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:42.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:42.030+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T22:18:42.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:42.431+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:42.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:42.431+0000 7fe57c841640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 msgr2=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:42.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:42.431+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:42.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:42.431+0000 7fe57c841640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 0x7fe5640190e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:42.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:42.431+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:42.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:42.431+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T22:18:43.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:43.232+0000 7fe57c841640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 msgr2=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3301/0 2024-03-28T22:18:43.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:43.232+0000 7fe57c841640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 0x7fe5640190e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:43.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:43.232+0000 7fe577fff640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.7:3300/0 2024-03-28T22:18:43.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:43.232+0000 7fe577fff640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:43.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:43.232+0000 7fe5777fe640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-28T22:18:43.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:43.232+0000 7fe5777fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T22:18:44.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:44.830+0000 7fe575ffb640 10 monclient(hunting): tick 2024-03-28T22:18:44.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:44.830+0000 7fe575ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-28T22:18:44.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:44.830+0000 7fe575ffb640 1 monclient(hunting): continuing hunt 2024-03-28T22:18:44.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:44.830+0000 7fe575ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-28T22:18:44.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:44.830+0000 7fe575ffb640 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 msgr2=0x7fe5640190e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:44.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:44.830+0000 7fe575ffb640 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe564014e40 0x7fe5640190e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:44.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:44.830+0000 7fe575ffb640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 msgr2=0x7fe564006510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:44.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:44.830+0000 7fe575ffb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe564008100 0x7fe564006510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:44.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:44.830+0000 7fe575ffb640 1 -- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 msgr2=0x7fe564014900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-28T22:18:44.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:44.830+0000 7fe575ffb640 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe564005a90 0x7fe564014900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T22:18:44.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:44.830+0000 7fe575ffb640 10 monclient: _add_conns ranks=[1,0] 2024-03-28T22:18:44.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:44.830+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe564019620 addr v1:172.21.15.83:6789/0 2024-03-28T22:18:44.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:44.830+0000 7fe575ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe564019a00 addr v1:172.21.15.7:6789/0 2024-03-28T22:18:44.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:44.830+0000 7fe575ffb640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780709c0 con 0x7fe564019a00 2024-03-28T22:18:44.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:44.830+0000 7fe575ffb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5781401b0 con 0x7fe564019620 2024-03-28T22:18:44.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:44.830+0000 7fe575ffb640 10 monclient(hunting): _renew_subs 2024-03-28T22:18:44.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:44.830+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564019a00 legacy=0x7fe56400cf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:44.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:44.830+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564019620 legacy=0x7fe5640073a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:45.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:45.031+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564019a00 legacy=0x7fe56400cf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:45.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:45.031+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564019620 legacy=0x7fe5640073a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:45.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:45.432+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564019a00 legacy=0x7fe56400cf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:45.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:45.432+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564019620 legacy=0x7fe5640073a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:46.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:46.233+0000 7fe5777fe640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7fe564019a00 legacy=0x7fe56400cf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-28T22:18:46.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-28T22:18:46.233+0000 7fe577fff640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7fe564019620 legacy=0x7fe5640073a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-28T22:18:47.750 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-28T22:18:47.751 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2764, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2596, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2362, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 1607, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-03-28T22:18:47.793 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-28T22:18:47.793 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-28T22:18:47.794 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-28T22:18:47.794 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T22:18:47.796 INFO:tasks.ceph.osd.0:Stopped 2024-03-28T22:18:47.796 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-28T22:18:47.796 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T22:18:47.797 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-28T22:18:47.797 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 1463, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2764, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2596, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2362, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 1607, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-28T22:18:47.807 INFO:tasks.ceph.osd.1:Stopped 2024-03-28T22:18:47.807 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-28T22:18:47.807 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T22:18:47.808 INFO:tasks.ceph.osd.2:Stopped 2024-03-28T22:18:47.808 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-28T22:18:47.808 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T22:18:47.808 INFO:tasks.ceph.osd.3:Stopped 2024-03-28T22:18:47.808 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-28T22:18:47.808 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T22:18:47.808 INFO:tasks.ceph.osd.4:Stopped 2024-03-28T22:18:47.809 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-28T22:18:47.809 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T22:18:47.809 INFO:tasks.ceph.osd.5:Stopped 2024-03-28T22:18:47.809 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-28T22:18:47.809 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T22:18:47.809 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-28T22:18:47.809 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 1463, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2764, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2596, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2362, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 1607, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi083 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-28T22:18:47.819 INFO:tasks.ceph.osd.6:Stopped 2024-03-28T22:18:47.820 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-28T22:18:47.820 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T22:18:47.820 INFO:tasks.ceph.osd.7:Stopped 2024-03-28T22:18:47.820 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-28T22:18:47.820 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-28T22:18:47.820 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T22:18:47.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:18:47.821 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 1463, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2764, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2596, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2362, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 1607, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y' 2024-03-28T22:18:47.831 INFO:tasks.ceph.mgr.y:Stopped 2024-03-28T22:18:47.831 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-28T22:18:47.831 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T22:18:47.831 INFO:tasks.ceph.mgr.x:Stopped 2024-03-28T22:18:47.831 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-28T22:18:47.831 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-28T22:18:47.832 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T22:18:47.832 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-28T22:18:47.832 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 1463, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2764, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2596, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2362, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 1607, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-28T22:18:47.842 INFO:tasks.ceph.mon.a:Stopped 2024-03-28T22:18:47.843 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-28T22:18:47.843 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T22:18:47.843 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-28T22:18:47.843 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 1463, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2764, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2596, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2362, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 1607, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-28T22:18:47.854 INFO:tasks.ceph.mon.c:Stopped 2024-03-28T22:18:47.854 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-28T22:18:47.854 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T22:18:47.854 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-28T22:18:47.854 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 1463, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2764, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2596, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2362, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 1607, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi083 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-28T22:18:47.865 INFO:tasks.ceph.mon.b:Stopped 2024-03-28T22:18:47.865 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-28T22:18:47.866 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-28T22:18:47.905 INFO:teuthology.orchestra.run.smithi007.stdout:2024-03-28T16:22:07.747113+0000 osd.2 (osd.2) 33 : cluster [WRN] osd.2 ep: 234 scrubber::ReplicaReservations pg[9.7]: timeout on replica reservations (since 2024-03-28 16:22:02) 2024-03-28T22:18:47.905 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-03-28T22:18:47.905 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-28T22:18:47.906 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-28T22:18:48.053 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-28T22:18:48.053 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-28T22:18:48.119 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-28T22:18:48.120 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-28T22:18:48.177 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-28T22:18:48.177 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-28T22:18:48.234 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-28T22:18:48.235 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-28T22:18:48.308 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-28T22:18:48.308 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-28T22:18:48.364 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-28T22:18:48.364 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-28T22:18:48.422 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-28T22:18:48.422 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-28T22:18:48.479 INFO:tasks.ceph:Archiving mon data... 2024-03-28T22:18:48.480 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-03-28_15:43:50-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7628301/data/mon.a.tgz 2024-03-28T22:18:48.481 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-03-28T22:18:48.498 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.waCBH01Y9W 2024-03-28T22:18:48.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.waCBH01Y9W 2024-03-28T22:18:48.733 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.waCBH01Y9W 2024-03-28T22:18:48.817 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.waCBH01Y9W is 674KB 2024-03-28T22:18:48.931 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.waCBH01Y9W 2024-03-28T22:18:48.945 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-03-28_15:43:50-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7628301/data/mon.c.tgz 2024-03-28T22:18:48.946 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-03-28T22:18:48.998 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.CvsElmPSSd 2024-03-28T22:18:48.999 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.CvsElmPSSd 2024-03-28T22:18:49.201 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.CvsElmPSSd 2024-03-28T22:18:49.286 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.CvsElmPSSd is 674KB 2024-03-28T22:18:49.389 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.CvsElmPSSd 2024-03-28T22:18:49.404 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-03-28_15:43:50-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7628301/data/mon.b.tgz 2024-03-28T22:18:49.404 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2024-03-28T22:18:49.419 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.qdNciXfkMY 2024-03-28T22:18:49.420 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.qdNciXfkMY 2024-03-28T22:18:49.637 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.qdNciXfkMY 2024-03-28T22:18:49.721 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.qdNciXfkMY is 674KB 2024-03-28T22:18:49.821 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.qdNciXfkMY 2024-03-28T22:18:49.836 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-28T22:18:49.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-28T22:18:49.839 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-28T22:18:49.901 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-03-28T22:18:49.901 DEBUG:teuthology.orchestra.run.smithi007:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-03-28T22:18:49.908 DEBUG:teuthology.orchestra.run.smithi083:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-03-28T22:18:49.949 INFO:teuthology.orchestra.run.smithi007.stdout:/var/log/ceph/valgrind/mon.a.log: Leak_StillReachable 2024-03-28T22:18:49.949 INFO:teuthology.orchestra.run.smithi007.stdout:/var/log/ceph/valgrind/mon.c.log: Leak_StillReachable 2024-03-28T22:18:49.949 INFO:teuthology.orchestra.run.smithi007.stdout:/var/log/ceph/valgrind/osd.1.log: Leak_PossiblyLost 2024-03-28T22:18:49.950 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.a.log kind Leak_StillReachable 2024-03-28T22:18:49.950 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.a.log 2024-03-28T22:18:49.954 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind Leak_StillReachable 2024-03-28T22:18:49.954 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.c.log 2024-03-28T22:18:49.959 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/osd.1.log kind Leak_PossiblyLost 2024-03-28T22:18:49.959 ERROR:tasks.ceph:saw valgrind issue Leak_PossiblyLost in /var/log/ceph/valgrind/osd.1.log 2024-03-28T22:18:49.981 INFO:teuthology.orchestra.run.smithi083.stdout:/var/log/ceph/valgrind/mon.b.log: Leak_StillReachable 2024-03-28T22:18:49.982 INFO:teuthology.orchestra.run.smithi083.stdout:/var/log/ceph/valgrind/osd.6.log: Leak_PossiblyLost 2024-03-28T22:18:49.982 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.b.log kind Leak_StillReachable 2024-03-28T22:18:49.982 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.b.log 2024-03-28T22:18:49.986 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/osd.6.log kind Leak_PossiblyLost 2024-03-28T22:18:49.986 ERROR:tasks.ceph:saw valgrind issue Leak_PossiblyLost in /var/log/ceph/valgrind/osd.6.log 2024-03-28T22:18:49.991 INFO:tasks.ceph:Archiving crash dumps... 2024-03-28T22:18:49.993 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-03-28_15:43:50-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7628301/remote/smithi007/crash 2024-03-28T22:18:49.994 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-28T22:18:50.091 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-03-28_15:43:50-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7628301/remote/smithi083/crash 2024-03-28T22:18:50.092 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-28T22:18:50.120 INFO:tasks.ceph:Compressing logs... 2024-03-28T22:18:50.120 DEBUG:teuthology.orchestra.run.smithi007:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T22:18:50.123 DEBUG:teuthology.orchestra.run.smithi083:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T22:19:06.659 INFO:tasks.ceph:Archiving logs... 2024-03-28T22:19:06.659 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-28_15:43:50-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7628301/remote/smithi007/log 2024-03-28T22:19:06.660 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T22:19:08.682 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-28_15:43:50-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7628301/remote/smithi083/log 2024-03-28T22:19:08.683 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T22:19:10.315 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 322, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2764, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2596, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2362, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 1607, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 1951, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 362, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2024-03-28T22:19:10.334 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-28T22:19:10.343 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 322, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2764, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2596, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2362, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 1607, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 1951, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 362, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2024-03-28T22:19:10.349 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-28T22:19:10.350 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T22:19:10.357 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T22:19:10.436 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-28T22:19:10.446 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-28T22:19:10.446 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T22:19:10.448 DEBUG:teuthology.orchestra.run.smithi083:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T22:19:10.462 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2024-03-28T22:19:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found 2024-03-28T22:19:10.466 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T22:19:10.466 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-03-28T22:19:10.466 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 31 -610us[ -616us] +/- 93ms 2024-03-28T22:19:10.467 INFO:teuthology.orchestra.run.smithi007.stdout:^* hv02.front.sepia.ceph.com 3 7 377 27 -779us[ -785us] +/- 69ms 2024-03-28T22:19:10.467 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +1780us[+1774us] +/- 82ms 2024-03-28T22:19:10.467 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T22:19:10.469 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T22:19:10.469 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-03-28T22:19:10.469 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 69 -620us[ -620us] +/- 92ms 2024-03-28T22:19:10.470 INFO:teuthology.orchestra.run.smithi083.stdout:^* hv02.front.sepia.ceph.com 3 9 377 174 -824us[ -974us] +/- 66ms 2024-03-28T22:19:10.470 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 +1777us[+1777us] +/- 82ms 2024-03-28T22:19:10.470 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T22:19:10.471 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-28T22:19:10.479 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-28T22:19:10.479 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-28T22:19:10.499 DEBUG:teuthology.orchestra.run.smithi007:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T22:19:10.579 DEBUG:teuthology.orchestra.run.smithi083:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T22:19:10.654 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-28T22:19:10.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:19:10.687 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-28T22:19:10.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:19:10.718 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-28T22:19:10.728 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-28T22:19:10.766 INFO:teuthology.task.internal:Duration was 22892.231621 seconds 2024-03-28T22:19:10.767 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-28T22:19:10.800 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-28T22:19:10.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T22:19:10.803 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T22:19:10.845 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T22:19:10.848 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T22:19:11.107 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-28T22:19:11.108 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi007.front.sepia.ceph.com 2024-03-28T22:19:11.108 DEBUG:teuthology.orchestra.run.smithi007:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T22:19:11.128 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi083.front.sepia.ceph.com 2024-03-28T22:19:11.128 DEBUG:teuthology.orchestra.run.smithi083:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T22:19:11.152 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-28T22:19:11.152 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T22:19:11.171 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T22:19:11.222 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-28T22:19:11.222 DEBUG:teuthology.orchestra.run.smithi007:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T22:19:11.241 DEBUG:teuthology.orchestra.run.smithi083:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T22:19:11.379 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-28T22:19:11.388 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-28T22:19:11.389 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T22:19:11.415 DEBUG:teuthology.orchestra.run.smithi083:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T22:19:11.443 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-28T22:19:11.453 DEBUG:teuthology.orchestra.run.smithi007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T22:19:11.458 DEBUG:teuthology.orchestra.run.smithi083:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T22:19:11.480 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = core 2024-03-28T22:19:11.509 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = core 2024-03-28T22:19:11.532 DEBUG:teuthology.orchestra.run.smithi007:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T22:19:11.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:19:11.568 DEBUG:teuthology.orchestra.run.smithi083:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T22:19:11.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:19:11.585 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-28T22:19:11.595 INFO:teuthology.task.internal:Transferring archived files... 2024-03-28T22:19:11.596 DEBUG:teuthology.misc:Transferring archived files from smithi007:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-28_15:43:50-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7628301/remote/smithi007 2024-03-28T22:19:11.596 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T22:19:11.645 DEBUG:teuthology.misc:Transferring archived files from smithi083:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-28_15:43:50-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7628301/remote/smithi083 2024-03-28T22:19:11.645 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T22:19:11.684 INFO:teuthology.task.internal:Removing archive directory... 2024-03-28T22:19:11.684 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T22:19:11.687 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T22:19:11.734 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-28T22:19:11.746 INFO:teuthology.task.internal:Not uploading archives. 2024-03-28T22:19:11.747 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-28T22:19:11.767 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-28T22:19:11.768 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T22:19:11.770 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T22:19:11.782 INFO:teuthology.orchestra.run.smithi007.stdout: 266024 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 28 22:19 /home/ubuntu/cephtest 2024-03-28T22:19:11.782 INFO:teuthology.orchestra.run.smithi007.stdout: 524861 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 28 16:15 /home/ubuntu/cephtest/mnt.0 2024-03-28T22:19:11.782 INFO:teuthology.orchestra.run.smithi007.stdout: 524862 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 28 16:16 /home/ubuntu/cephtest/mnt.0/client.0 2024-03-28T22:19:11.783 INFO:teuthology.orchestra.run.smithi007.stdout: 669247 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 16:16 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T22:19:11.783 INFO:teuthology.orchestra.run.smithi007.stdout: 669258 4 -rw-r--r-- 1 ubuntu ubuntu 1214 Mar 28 16:28 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_tier_pp.log 2024-03-28T22:19:11.783 INFO:teuthology.orchestra.run.smithi007.stdout: 669263 4 -rw-r--r-- 1 ubuntu ubuntu 1278 Mar 28 16:19 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_stat_pp.log 2024-03-28T22:19:11.783 INFO:teuthology.orchestra.run.smithi007.stdout: 669253 4 -rw-r--r-- 1 ubuntu ubuntu 1872 Mar 28 16:19 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_lock.log 2024-03-28T22:19:11.783 INFO:teuthology.orchestra.run.smithi007.stdout: 669272 4 -rw-r--r-- 1 ubuntu ubuntu 1714 Mar 28 16:18 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_c_read_operations.log 2024-03-28T22:19:11.783 INFO:teuthology.orchestra.run.smithi007.stdout: 669273 8 -rw-r--r-- 1 ubuntu ubuntu 5573 Mar 28 16:20 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_list_parallel.log 2024-03-28T22:19:11.783 INFO:teuthology.orchestra.run.smithi007.stdout: 669256 8 -rw-r--r-- 1 ubuntu ubuntu 4276 Mar 28 16:25 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_misc_pp.log 2024-03-28T22:19:11.783 INFO:teuthology.orchestra.run.smithi007.stdout: 669248 4 -rw-r--r-- 1 ubuntu ubuntu 4047 Mar 28 16:26 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_aio.log 2024-03-28T22:19:11.784 INFO:teuthology.orchestra.run.smithi007.stdout: 669255 4 -rw-r--r-- 1 ubuntu ubuntu 2591 Mar 28 16:19 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_misc.log 2024-03-28T22:19:11.784 INFO:teuthology.orchestra.run.smithi007.stdout: 669271 4 -rw-r--r-- 1 ubuntu ubuntu 423 Mar 28 16:17 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_cls_remote_reads.log 2024-03-28T22:19:11.784 INFO:teuthology.orchestra.run.smithi007.stdout: 669266 4 -rw-r--r-- 1 ubuntu ubuntu 591 Mar 28 16:18 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_cmd_pp.log 2024-03-28T22:19:11.784 INFO:teuthology.orchestra.run.smithi007.stdout: 669265 20 -rw-r--r-- 1 ubuntu ubuntu 16552 Mar 28 16:19 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_cmd.log 2024-03-28T22:19:11.784 INFO:teuthology.orchestra.run.smithi007.stdout: 669264 4 -rw-r--r-- 1 ubuntu ubuntu 2721 Mar 28 16:19 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_watch_notify.log 2024-03-28T22:19:11.784 INFO:teuthology.orchestra.run.smithi007.stdout: 669260 4 -rw-r--r-- 1 ubuntu ubuntu 2073 Mar 28 16:20 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_snapshots.log 2024-03-28T22:19:11.784 INFO:teuthology.orchestra.run.smithi007.stdout: 669252 128 -rw-r--r-- 1 ubuntu ubuntu 130395 Mar 28 16:17 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_asio.log 2024-03-28T22:19:11.784 INFO:teuthology.orchestra.run.smithi007.stdout: 669262 4 -rw-r--r-- 1 ubuntu ubuntu 1175 Mar 28 16:18 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_stat.log 2024-03-28T22:19:11.784 INFO:teuthology.orchestra.run.smithi007.stdout: 669274 4 -rw-r--r-- 1 ubuntu ubuntu 1295 Mar 28 16:19 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_delete_pools_parallel.log 2024-03-28T22:19:11.785 INFO:teuthology.orchestra.run.smithi007.stdout: 669270 4 -rw-r--r-- 1 ubuntu ubuntu 713 Mar 28 16:17 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_service_pp.log 2024-03-28T22:19:11.785 INFO:teuthology.orchestra.run.smithi007.stdout: 669269 4 -rw-r--r-- 1 ubuntu ubuntu 1204 Mar 28 16:19 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_c_write_operations.log 2024-03-28T22:19:11.785 INFO:teuthology.orchestra.run.smithi007.stdout: 669268 4 -rw-r--r-- 1 ubuntu ubuntu 2687 Mar 28 16:17 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_service.log 2024-03-28T22:19:11.785 INFO:teuthology.orchestra.run.smithi007.stdout: 669249 8 -rw-r--r-- 1 ubuntu ubuntu 4693 Mar 28 16:26 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_aio_pp.log 2024-03-28T22:19:11.785 INFO:teuthology.orchestra.run.smithi007.stdout: 669251 4 -rw-r--r-- 1 ubuntu ubuntu 3900 Mar 28 16:19 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_io_pp.log 2024-03-28T22:19:11.785 INFO:teuthology.orchestra.run.smithi007.stdout: 669275 4 -rw-r--r-- 1 ubuntu ubuntu 1029 Mar 28 16:19 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_open_pools_parallel.log 2024-03-28T22:19:11.786 INFO:teuthology.orchestra.run.smithi007.stdout: 669267 8 -rw-r--r-- 1 ubuntu ubuntu 4139 Mar 28 16:19 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_watch_notify_pp.log 2024-03-28T22:19:11.786 INFO:teuthology.orchestra.run.smithi007.stdout: 669257 4 -rw-r--r-- 1 ubuntu ubuntu 2020 Mar 28 16:19 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_lock_pp.log 2024-03-28T22:19:11.786 INFO:teuthology.orchestra.run.smithi007.stdout: 669259 4 -rw-r--r-- 1 ubuntu ubuntu 1241 Mar 28 16:17 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_pool.log 2024-03-28T22:19:11.786 INFO:teuthology.orchestra.run.smithi007.stdout: 669254 8 -rw-r--r-- 1 ubuntu ubuntu 5479 Mar 28 16:25 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_list.log 2024-03-28T22:19:11.786 INFO:teuthology.orchestra.run.smithi007.stdout: 669261 4 -rw-r--r-- 1 ubuntu ubuntu 3791 Mar 28 16:23 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_snapshots_pp.log 2024-03-28T22:19:11.786 INFO:teuthology.orchestra.run.smithi007.stdout: 669250 4 -rw-r--r-- 1 ubuntu ubuntu 2506 Mar 28 16:19 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_io.log 2024-03-28T22:19:11.786 INFO:teuthology.orchestra.run.smithi007.stdout: 277198 720132 -rw------- 1 root root 737841152 Mar 28 16:29 /home/ubuntu/cephtest/vgcore.34919 2024-03-28T22:19:11.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T22:19:11.787 INFO:teuthology.orchestra.run.smithi007.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-28T22:19:11.788 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 322, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 1944, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2764, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2596, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 2362, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph_manager.py", line 1607, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 1951, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/ceph.py", line 362, in valgrind_post raise valgrind_exception Exception: saw valgrind issues During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-28T22:19:11.809 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-28T22:19:11.854 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-28T22:19:11.865 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: saw valgrind issues 2024-03-28T22:19:11.865 INFO:teuthology.nuke:Checking targets against current locks 2024-03-28T22:19:11.878 INFO:teuthology.orchestra.run.smithi083.stdout: 265844 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 22:19 /home/ubuntu/cephtest 2024-03-28T22:19:11.879 INFO:teuthology.orchestra.run.smithi083.stdout: 267175 830748 -rw------- 1 root root 851144704 Mar 28 16:26 /home/ubuntu/cephtest/vgcore.35297 2024-03-28T22:19:11.879 INFO:teuthology.orchestra.run.smithi083.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-28T22:19:11.897 DEBUG:teuthology.nuke:shortname: smithi007 2024-03-28T22:19:11.898 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T22:19:11.917 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-28_15:43:50-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7628301', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 15:49:09.915584', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPSlIhOqcL4HPPLuorlvP91WM/eGDMhSxwDVQxUeX+rJZctwWzPZuqCmEEZMlGLS1+H4aGa0KHzKTPaqWlGLt5A='} 2024-03-28T22:19:11.921 DEBUG:teuthology.nuke:shortname: smithi083 2024-03-28T22:19:11.921 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T22:19:11.938 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-28_15:43:50-rados-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi/7628301', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 15:49:09.914228', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPEEzb0wEA3mhYdRzG0yLCtBdbNckMpzYgietdNUhBv07DwgXUIwEb937mdiXay9pYyp/gyhuKgZaVKV1FcQxNA='} 2024-03-28T22:19:11.967 INFO:teuthology.orchestra.console.smithi007:Power off 2024-03-28T22:19:11.968 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T22:19:11.997 INFO:teuthology.orchestra.console.smithi083:Power off 2024-03-28T22:19:11.997 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T22:19:12.008 DEBUG:teuthology.orchestra.console.smithi007:power off output: Chassis Power Control: Down/Off 2024-03-28T22:19:12.009 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:19:12.022 DEBUG:teuthology.orchestra.console.smithi083:power off output: Chassis Power Control: Down/Off 2024-03-28T22:19:12.023 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:19:12.032 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-03-28T22:19:12.045 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2024-03-28T22:19:16.034 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:19:16.046 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:19:16.145 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-03-28T22:19:16.154 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2024-03-28T22:19:20.149 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:19:20.158 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T22:19:20.261 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is off 2024-03-28T22:19:20.269 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is off 2024-03-28T22:19:20.362 INFO:teuthology.orchestra.console.smithi007:Power off completed 2024-03-28T22:19:20.370 INFO:teuthology.orchestra.console.smithi083:Power off completed 2024-03-28T22:19:20.476 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} duration: 22892.231620550156 failure_reason: 'Command failed (workunit test rados/test.sh) on smithi007 with status 124: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e58b9979847f029f7ab1514b5f5f99b9f03d97f6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6eaa52a0fd0a46f8b253f613d5b82baf status: fail success: false 2024-03-28T22:19:20.477 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T22:19:20.564 INFO:teuthology.run:FAIL